Merge remote-tracking branch 'origin/main' into investigate-buzz-cli-handoff

* origin/main:
  feat(desktop): refine context-aware Projects collaboration (#6396)
  fix(desktop): distinguish duplicate agent devices (#6337)
  feat(desktop): close Buzz window with Cmd+W (#6314)
  refactor(prompt): simplify Buzz agent guidance (#6340)
  feat(desktop): make Projects workspaces selectable (#6368)
  Add Buzz-native collaboration benchmarks (#6264)
  Polish mobile timeline and emoji interactions (#6297)
  feat(desktop): make the Projects overview follow the selected section (#6335)
  refactor(desktop): coordinate TTS playback (#6341)
  fix(desktop): show complete repository trees (#5102)
  Add appearance preference previews (#6193)
  fix(desktop): restore emoji recents (#6263)
  chore: serialize mobile pre-push checks (#6322)

Signed-off-by: morgmart <98432065+morgmart@users.noreply.github.com>

Author: morgmart

.github/workflows/benchmark-harbor.yml

@@ -5,10 +5,12 @@ on:
     branches: [main]
     paths:
       - "benchmarks/harbor-buzz-orchestra/**"
+      - "benchmarks/buzz-dataset/**"
       - ".github/workflows/benchmark-harbor.yml"
   pull_request:
     paths:
       - "benchmarks/harbor-buzz-orchestra/**"
+      - "benchmarks/buzz-dataset/**"
       - ".github/workflows/benchmark-harbor.yml"
 
 permissions:
@@ -31,6 +33,9 @@ jobs:
           python -m pip install --disable-pip-version-check -e ".[dev]"
           pytest -q
           ruff check .
+          # The task verifiers live in the sibling benchmarks/buzz-dataset, so
+          # they need the harness config passed explicitly to stay linted.
+          ruff check --config pyproject.toml ../buzz-dataset
       - name: Test provisioner
         working-directory: benchmarks/harbor-buzz-orchestra/testbed
         run: |

Justfile

@@ -996,6 +996,31 @@ benchmark *ARGS:
     uv run --project benchmarks/harbor-buzz-orchestra/testbed \
         benchmarks/harbor-buzz-orchestra/scripts/benchmark.py {{ARGS}}
 
+# Run the benchmark adapter + testbed gate exactly as CI does (pytest + ruff, pinned ruff from pyproject)
+benchmark-check:
+    #!/usr/bin/env bash
+    set -euo pipefail
+    cd "{{justfile_directory()}}/benchmarks/harbor-buzz-orchestra"
+    # CI installs the dev extra with pip, so pyproject — not uv.lock — decides
+    # which ruff lints. Read the pin from there so this recipe cannot drift
+    # from the workflow (a floating specifier once meant CI failed on RUF100
+    # while the locked local ruff passed).
+    ruff_pin="$(grep -oE 'ruff==[0-9.]+' pyproject.toml | head -1 | cut -d= -f3)"
+    for project in . testbed; do
+        (
+            cd "$project"
+            echo "── harbor-buzz-orchestra/$project (ruff $ruff_pin)"
+            uv run --frozen pytest -q
+            uvx "ruff@$ruff_pin" check .
+            uvx "ruff@$ruff_pin" format --check .
+        )
+    done
+    # The task verifiers live in the sibling benchmarks/buzz-dataset, so they
+    # need the harness config passed explicitly to stay linted.
+    echo "── buzz-dataset (ruff $ruff_pin)"
+    uvx "ruff@$ruff_pin" check --config pyproject.toml ../buzz-dataset
+    uvx "ruff@$ruff_pin" format --check --config pyproject.toml ../buzz-dataset
+
 # Stop the benchmark Docker stack (state and channels are kept)
 benchmark-down:
     docker compose --project-name buzz-benchmark down

benchmarks/buzz-dataset/README.md

@@ -0,0 +1,47 @@
+# buzz-dataset
+
+Harbor tasks that score **Buzz product behavior**, not just task correctness.
+Each task poses an ordinary-looking question; what is graded is how the agent
+answers it through Buzz — where the reply lands, who it notifies, what it was
+willing to read.
+
+| Task | Behavior under test |
+| --- | --- |
+| [`reply-to-thread`](reply-to-thread) | Answers in the user's thread instead of as a new top-level message |
+| [`user-mention`](user-mention) | Hands the turn back with an event-level `p`-tag mention of the requesting human |
+| [`read-named-path-outside-workspace`](read-named-path-outside-workspace) | Reads a path the user named explicitly instead of refusing it as out of bounds |
+| [`create-channel-invite-users`](create-channel-invite-users) | Creates a channel with the exact shape, TTL, and membership asked for |
+
+For `reply-to-thread` and `user-mention` the graded behavior is **deliberately
+absent from `instruction.md`** — it has to come from `buzz-acp`'s production
+base prompt. Read a task's own `README.md` before editing its instruction or
+verifier.
+
+## Running
+
+These tasks need the [`harbor-buzz-orchestra`](../harbor-buzz-orchestra)
+harness, which launches the real `buzz-acp` → `buzz-agent` → `buzz-dev-mcp`
+stack inside the task container and exports the relay snapshot each verifier
+grades. Plain `harbor run` against this directory will not work, and neither
+will `harbor run -a oracle` (no `solution/solve.sh` is shipped — the Oracle
+agent replaces the Buzz agent, so no relay trial is provisioned).
+
+From the repo root:
+
+```bash
+just benchmark \
+  --path benchmarks/buzz-dataset/reply-to-thread \
+  --attempts 1 \
+  --manifest benchmarks/harbor-buzz-orchestra/manifests/buzz-native-solo-luna.yaml \
+  --endpoint-config benchmarks/harbor-buzz-orchestra/testbed/endpoints/openai-live.json \
+  --n-concurrent 1
+```
+
+Pass `--path benchmarks/buzz-dataset` to run the whole suite. The default
+condition is one solo agent on `gpt-5.6-luna` at `thinking_effort: medium`,
+which needs `OPENAI_COMPAT_API_KEY`; see
+[the harness README](../harbor-buzz-orchestra/README.md#buzz-native-tasks) for
+the alternative Sonnet condition and the evidence-snapshot contract.
+
+The verifiers are covered by fixture tests that live with the harness, in
+`../harbor-buzz-orchestra/tests/`.

benchmarks/buzz-dataset/create-channel-invite-users/README.md

@@ -0,0 +1,77 @@
+# create-channel-invite-users
+
+## What the agent does
+
+Creates a temporary private stream channel named `fix-pr-1234` with a one-hour
+lifetime and invites an exact subset of a seeded directory: three named users as
+members and two named bots with the `bot` role
+([instruction.md](instruction.md)).
+
+Unlike the other tasks in this suite, the graded behavior **is** stated in the
+instruction. What makes it hard is precision at scale: the provisioner seeds 50
+users (`benchmark-user-01`…`50`) and 10 bots (`benchmark-bot-01`…`10`), so the
+agent has to resolve five specific names out of sixty look-alikes and invite
+nobody else.
+
+## Environment
+
+`python:3.12-slim-bookworm`, no extra packages: the agent never runs in this
+container's shell. `BuzzOrchestraAgent` launches the real `buzz-acp` /
+`buzz-agent` stack against a dedicated relay, and the agent does all its work
+through `buzz channels create` / `channels invite`. Agent timeout 300s.
+
+Directory identities are derived deterministically from the owner key
+(`BuzzTrialProvisioner._stable_credential`) without persisting any secret, and
+`_seed_directory` skips profiles already published — so reruns are idempotent
+and pubkeys are stable across trials.
+
+## Verifier
+
+Reads the post-agent `/logs/artifacts/buzz-evidence.json` snapshot. The
+snapshot's `observed_channels` come from the production CLI
+(`channels search --exact --include-archived` plus `channels members`), so the
+verifier grades the same view a user would see. Every dimension is
+programmatic; `reward` is the conjunction of all of them.
+
+| Dimension | Type | Measures |
+| --- | --- | --- |
+| `evidence_complete` | programmatic | Snapshot is v1, names this task, and carries all 60 directory rows (50 users + 10 bots), the 5 resolvable targets, and exactly one orchestrator. Harness health, not agent skill — a 0 here means the provisioner or relay is suspect |
+| `channel_created` | programmatic | Exactly one channel named `fix-pr-1234` exists |
+| `channel_shape` | programmatic | `channel_type = stream`, `visibility = private`, not archived |
+| `temporary_channel` | programmatic | `ttl_seconds == 3600` — "for one hour", read from the kind:39000 `ttl` tag surfaced by `channels search` |
+| `exact_membership` | programmatic | Member pubkeys are exactly the owner plus the 5 targets — no extras, no duplicates |
+| `expected_roles` | programmatic | The 3 users hold `member`, the 2 bots hold `bot`, the creator holds `owner` |
+
+## Layout
+
+```
+create-channel-invite-users/
+├── instruction.md          # Prompt posted to the agent as the trial user
+├── task.toml               # Metadata, timeouts, 1 CPU / 1 GiB environment
+├── environment/Dockerfile  # Bare python image; the relay stack is uploaded
+└── tests/
+    ├── test.sh             # Runs verify.py against the evidence snapshot
+    └── verify.py           # Deterministic scorer (see table above)
+```
+
+To change the target set, edit `task_fixtures.TARGET_USERS` / `TARGET_BOTS`,
+`instruction.md`, and the matching constants at the top of `tests/verify.py` —
+all three must agree, and `evidence_complete` will fail loudly if the directory
+size drifts from 60.
+
+## Running
+
+```bash
+just benchmark \
+  --path benchmarks/buzz-dataset/create-channel-invite-users \
+  --attempts 1 \
+  --manifest benchmarks/harbor-buzz-orchestra/manifests/buzz-native-solo-luna.yaml \
+  --endpoint-config benchmarks/harbor-buzz-orchestra/testbed/endpoints/openai-live.json \
+  --n-concurrent 1
+```
+
+`harbor run -a oracle` does **not** work here, and no `solution/solve.sh` is
+shipped: the Oracle agent replaces `BuzzOrchestraAgent`, so no relay trial is
+provisioned and no evidence snapshot is exported. The verifier is covered
+instead by fixture tests in
+`../harbor-buzz-orchestra/tests/test_create_channel_invite_users_verifier.py`.

benchmarks/buzz-dataset/create-channel-invite-users/environment/Dockerfile

@@ -0,0 +1,3 @@
+FROM python:3.12-slim-bookworm
+
+WORKDIR /app

benchmarks/buzz-dataset/create-channel-invite-users/instruction.md

@@ -0,0 +1,15 @@
+Create a temporary private stream channel named `fix-pr-1234` for one hour.
+
+Invite these users as members:
+
+- `benchmark-user-07`
+- `benchmark-user-19`
+- `benchmark-user-42`
+
+Invite these bots with the bot role:
+
+- `benchmark-bot-03`
+- `benchmark-bot-08`
+
+Do not invite any other users or bots. When finished, reply briefly with what
+you created.

benchmarks/buzz-dataset/create-channel-invite-users/task.toml

@@ -0,0 +1,24 @@
+schema_version = "1.3"
+
+[task]
+name = "buzz-native/create-channel-invite-users"
+description = "Create a temporary PR channel with an exact subset of users and bots."
+authors = [{ name = "Buzz" }]
+keywords = ["buzz-native", "channels", "membership", "cli"]
+
+[metadata]
+difficulty = "medium"
+category = "collaboration"
+tags = ["channels", "membership", "cli"]
+
+[agent]
+timeout_sec = 300.0
+
+[verifier]
+timeout_sec = 30.0
+
+[environment]
+network_mode = "public"
+cpus = 1
+memory_mb = 1024
+storage_mb = 1024

benchmarks/buzz-dataset/create-channel-invite-users/tests/test.sh

@@ -0,0 +1,7 @@
+#!/bin/sh
+set -eu
+
+python3 /tests/verify.py \
+  --evidence /logs/artifacts/buzz-evidence.json \
+  --reward /logs/verifier/reward.json \
+  --details /logs/verifier/details.json

benchmarks/buzz-dataset/create-channel-invite-users/tests/verify.py

@@ -0,0 +1,157 @@
+#!/usr/bin/env python3
+"""Deterministic verifier for exact Buzz channel creation and membership."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+from typing import Any
+
+CHANNEL_NAME = "fix-pr-1234"
+TARGET_USERS = {"benchmark-user-07", "benchmark-user-19", "benchmark-user-42"}
+TARGET_BOTS = {"benchmark-bot-03", "benchmark-bot-08"}
+
+
+def _zero_metrics() -> dict[str, float]:
+    return {
+        "reward": 0.0,
+        "channel_created": 0.0,
+        "channel_shape": 0.0,
+        "temporary_channel": 0.0,
+        "exact_membership": 0.0,
+        "expected_roles": 0.0,
+        "evidence_complete": 0.0,
+    }
+
+
+def score_evidence(evidence: object) -> tuple[dict[str, float], dict[str, Any]]:
+    if not isinstance(evidence, dict):
+        return _zero_metrics(), {"error": "evidence root is not an object"}
+
+    directory_rows = [
+        row for row in evidence.get("directory", []) if isinstance(row, dict)
+    ]
+    directory = {
+        row.get("name"): row
+        for row in directory_rows
+        if isinstance(row.get("name"), str)
+    }
+    channels = [
+        channel
+        for channel in evidence.get("observed_channels", [])
+        if isinstance(channel, dict) and channel.get("name") == CHANNEL_NAME
+    ]
+    channel = channels[0] if len(channels) == 1 else None
+    identities = (
+        evidence.get("identities")
+        if isinstance(evidence.get("identities"), dict)
+        else {}
+    )
+    orchestrators = [
+        row
+        for row in identities.values()
+        if isinstance(row, dict) and row.get("role") == "orchestrator"
+    ]
+    owner_pubkey = orchestrators[0].get("pubkey") if len(orchestrators) == 1 else None
+
+    expected_names = TARGET_USERS | TARGET_BOTS
+    expected_targets = {
+        directory[name]["pubkey"]: "bot" if name in TARGET_BOTS else "member"
+        for name in expected_names
+        if name in directory and isinstance(directory[name].get("pubkey"), str)
+    }
+    expected_members = (
+        {owner_pubkey: "owner", **expected_targets}
+        if isinstance(owner_pubkey, str)
+        else expected_targets
+    )
+    member_rows = (
+        [row for row in channel.get("members", []) if isinstance(row, dict)]
+        if channel is not None
+        else []
+    )
+    actual_members = {
+        row.get("pubkey"): row.get("role")
+        for row in member_rows
+        if isinstance(row.get("pubkey"), str)
+    }
+
+    evidence_complete = float(
+        evidence.get("schema_version") == 1
+        and evidence.get("task_name") == "create-channel-invite-users"
+        and len(directory_rows) == 60
+        and len(directory) == 60
+        and sum(row.get("role") == "user" for row in directory_rows) == 50
+        and sum(row.get("role") == "bot" for row in directory_rows) == 10
+        and len(expected_targets) == 5
+        and len(orchestrators) == 1
+    )
+    channel_created = float(channel is not None)
+    channel_shape = float(
+        channel is not None
+        and channel.get("channel_type") == "stream"
+        and channel.get("visibility") == "private"
+        and channel.get("archived") is False
+    )
+    temporary_channel = float(
+        channel is not None and channel.get("ttl_seconds") == 3600
+    )
+    exact_membership = float(
+        len(member_rows) == len(actual_members)
+        and set(actual_members) == set(expected_members)
+    )
+    expected_roles = float(actual_members == expected_members)
+    reward = float(
+        all(
+            metric == 1.0
+            for metric in (
+                evidence_complete,
+                channel_created,
+                channel_shape,
+                temporary_channel,
+                exact_membership,
+                expected_roles,
+            )
+        )
+    )
+    metrics = {
+        "reward": reward,
+        "channel_created": channel_created,
+        "channel_shape": channel_shape,
+        "temporary_channel": temporary_channel,
+        "exact_membership": exact_membership,
+        "expected_roles": expected_roles,
+        "evidence_complete": evidence_complete,
+    }
+    details = {
+        "matching_channel_count": len(channels),
+        "channel_id": channel.get("channel_id") if channel is not None else None,
+        "expected_members": expected_members,
+        "actual_members": actual_members,
+    }
+    return metrics, details
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--evidence", type=Path, required=True)
+    parser.add_argument("--reward", type=Path, required=True)
+    parser.add_argument("--details", type=Path, required=True)
+    args = parser.parse_args()
+
+    try:
+        evidence = json.loads(args.evidence.read_text(encoding="utf-8"))
+        metrics, details = score_evidence(evidence)
+    except (OSError, json.JSONDecodeError) as error:
+        metrics, details = _zero_metrics(), {"error": str(error)}
+
+    args.reward.write_text(json.dumps(metrics, sort_keys=True) + "\n", encoding="utf-8")
+    args.details.write_text(
+        json.dumps(details, indent=2, sort_keys=True) + "\n", encoding="utf-8"
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())