Vally: fix MCP boot race + drop misconfigured grader (#15948)

- Launch pre-built DLLs via 'dotnet <dll>' in both .vally.yaml files instead of 'dotnet run', so N parallel workers no longer race on Roslyn's exclusive write lock for the output DLL.

- Add 'Build MCP servers' step to eng/pipelines/skill-eval.yml so the CI runner has the DLLs ready before vally starts.

- Drop the skill-invocation grader from generate-sdk-for-existing-release-plan (no preflight reasoning step required; tools-only).

- Strip 'I'm in a checkout of azure-rest-api-specs.' preamble from prompts; the worktree already provides that context.

- Remove stray '// tools skills response' artifact in live release-planner.eval.yaml.

- README: document 'dotnet build' as a prereq; rewrite workers warning.

Validated: scenarios-mock at --workers 6 -> 5/5 stimuli pass, 0 race hits, ~4 min.

Author: helen229

.github/skills/.vally.yaml

@@ -12,12 +12,15 @@ paths:
   evalFilenames: ["eval.yaml", "*.eval.yaml"]
 
 environments:
+  # Launch the pre-built DLLs via `dotnet <dll>`, NOT `dotnet run` — avoids the
+  # MSBuild boot race under parallel workers. See issue #15948.
+  # CI builds the DLLs in the 'Build MCP servers' step of skill-eval.yml.
   azsdk-mcp:
     mcpServers:
       azure-sdk-mcp:
         type: stdio
         command: dotnet
-        args: ["run", "--project", "../../tools/azsdk-cli/Azure.Sdk.Tools.Cli", "--", "start"]
+        args: ["../../artifacts/bin/Azure.Sdk.Tools.Cli/Debug/net8.0/azsdk.dll", "start"]
         timeout: "60s"
         env:
           AZSDKTOOLS_AGENT_TESTING: "true"
@@ -27,5 +30,5 @@ environments:
       azure-sdk-mcp:
         type: stdio
         command: dotnet
-        args: ["run", "--project", "../../tools/azsdk-cli/Azure.Sdk.Tools.Mock"]
+        args: ["../../artifacts/bin/Azure.Sdk.Tools.Mock/Debug/net8.0/azsdk-mock.dll"]
         timeout: "60s"

eng/pipelines/skill-eval.yml

@@ -41,6 +41,14 @@ jobs:
       - script: npm install -g @github/copilot-sdk
         displayName: 'Install Copilot SDK'
 
+      # Pre-build the MCP servers so vally launches `dotnet <dll>` instead of
+      # `dotnet run` — avoids the MSBuild boot race under parallel workers.
+      # See issue #15948.
+      - script: |
+          dotnet build tools/azsdk-cli/Azure.Sdk.Tools.Cli  -c Debug --nologo
+          dotnet build tools/azsdk-cli/Azure.Sdk.Tools.Mock -c Debug --nologo
+        displayName: 'Build MCP servers'
+
       - script: |
           input_areas=$(echo "${{ parameters.areas }}" | xargs)
           if [ -n "$input_areas" ]; then

tools/azsdk-cli/Azure.Sdk.Tools.Vally/.vally.yaml

@@ -11,33 +11,26 @@ paths:
   results: results/
 
 environments:
-  # Default for unit + mock scenarios. Runs the dedicated Azure.Sdk.Tools.Mock
-  # MCP server — a separate process whose tool surface mirrors the real CLI
-  # but with deterministic in-memory responses.
-  #
-  # Relative `--project` paths are resolved by `dotnet` against the cwd of
-  # the vally invocation. Always run vally from this directory:
-  #   cd tools/azsdk-cli/Azure.Sdk.Tools.Vally && vally eval ...
-  # Same convention as .github/skills/.vally.yaml.
+  # Launch the pre-built DLL via `dotnet <dll>`, NOT `dotnet run` — avoids the
+  # MSBuild boot race under parallel workers. See issue #15948.
+  # Run `dotnet build ../Azure.Sdk.Tools.Mock -c Debug` once before vally.
   azsdk-mcp-mock:
     mcpServers:
       azure-sdk-mcp:
         type: stdio
         command: dotnet
-        args: ["run", "--project", "../Azure.Sdk.Tools.Mock"]
-        timeout: "60s"
+        args: ["../../../artifacts/bin/Azure.Sdk.Tools.Mock/Debug/net8.0/azsdk-mock.dll"]
+        timeout: "30s"
 
-  # Live MCP — real Azure.Sdk.Tools.Cli against real DevOps (test area path),
-  # real GitHub, real pipelines. AZSDKTOOLS_AGENT_TESTING=true keeps the
-  # handful of write tools (e.g. create_release_plan) inside the test area.
-  # Bound only by scenarios under evals/workflow-scenarios/live/ and selected
-  # by the `scenarios-live` / `nightly` suites.
+  # Live MCP. AZSDKTOOLS_AGENT_TESTING=true keeps write tools inside the test
+  # area. Pre-built DLL pattern — see issue #15948.
+  # Run `dotnet build ../Azure.Sdk.Tools.Cli -c Debug` once before vally.
   azsdk-mcp-live:
     mcpServers:
       azure-sdk-mcp:
         type: stdio
         command: dotnet
-        args: ["run", "--project", "../Azure.Sdk.Tools.Cli", "--", "start"]
+        args: ["../../../artifacts/bin/Azure.Sdk.Tools.Cli/Debug/net8.0/azsdk.dll", "start"]
         timeout: "5m"
         env:
           AZSDKTOOLS_AGENT_TESTING: "true"

tools/azsdk-cli/Azure.Sdk.Tools.Vally/README.md

@@ -144,6 +144,19 @@ Prereqs:
   cd eng/skill-eval
   npm ci
   ```
+- **Build the MCP servers once** before running vally. `.vally.yaml`
+  launches the pre-built DLLs via `dotnet <dll>` to avoid the build-time
+  race that crashes parallel workers with `MCP error -32000: Connection
+  closed`. See [`primer-vally-mcp-race.html`](https://github.com/Azure/azure-sdk-tools/blob/main/doc/) (notebook) for the full write-up.
+
+  ```powershell
+  # From repo root — builds both Azure.Sdk.Tools.Mock and (transitively) Cli
+  dotnet build tools/azsdk-cli/Azure.Sdk.Tools.Cli -c Debug
+  dotnet build tools/azsdk-cli/Azure.Sdk.Tools.Mock -c Debug
+  ```
+
+  Rebuild after editing any tool source. Vally itself does **not** rebuild
+  the MCP server — it just spawns the existing DLL.
 
 Run a suite (recommended):
 
@@ -168,11 +181,12 @@ $skills = '../../../.github/skills'
 > the agent never loads the project skills and the `skill-invocation`
 > grader fails even when the tool calls are correct.
 >
-> Each agent boots its own MCP child process, so parallel workers compete
-> for stdio startup. Keep `--workers` at 1–2 for `scenarios-mock` / live
-> runs until we share a single Mock MCP server across workers — higher
-> concurrency triggers `MCP server 'azure-sdk-mcp' failed to load:
-> Connection closed` on most stimuli.
+> Each agent still boots its own MCP child process, but `.vally.yaml`
+> launches the **pre-built** `azsdk-mock.dll` / `azsdk.dll` via
+> `dotnet <dll>` (read-only memory-map, no MSBuild on the hot path), so
+> `--workers 6+` is safe for `scenarios-mock`. The old MSBuild boot race
+> is gone; the only remaining concurrency limit is rate limits on the
+> Copilot CLI subprocesses.
 
 Run a single eval:
 

tools/azsdk-cli/Azure.Sdk.Tools.Vally/evals/workflow-scenarios/live/release-planner.eval.yaml

@@ -63,7 +63,7 @@ stimuli:
         source: C:/Users/gaoh/.vally-cache/azure-rest-api-specs
         ref: main
     prompt: |
-      I'm in a checkout of azure-rest-api-specs. Walk me through the full
+      Walk me through the full
       release-plan + SDK-generation flow for the Contoso Widget Manager
       end-to-end. Do every step below, in order, and use real tools (no
       dry-run, no simulation):

tools/azsdk-cli/Azure.Sdk.Tools.Vally/evals/workflow-scenarios/mock/release-planner-workflows.eval.yaml

@@ -44,7 +44,7 @@ stimuli:
         source: C:/Users/gaoh/.vally-cache/azure-rest-api-specs
         ref: main
     prompt: |
-      I'm in a checkout of azure-rest-api-specs. Create a public preview
+      Create a public preview
       release plan for the Contoso Widget Manager. Here is all the context
       you need:
         - TypeSpec project: "specification/contosowidgetmanager/Contoso.WidgetManager"
@@ -81,7 +81,7 @@ stimuli:
         source: C:/Users/gaoh/.vally-cache/azure-rest-api-specs
         ref: main
     prompt: |
-      I'm in a checkout of azure-rest-api-specs. Walk me through creating
+      Walk me through creating
       a release plan and then generating SDK for the Contoso Widget Manager:
         - TypeSpec project: "specification/contosowidgetmanager/Contoso.WidgetManager"
         - API release type: "Public Preview"
@@ -125,7 +125,7 @@ stimuli:
         source: C:/Users/gaoh/.vally-cache/azure-rest-api-specs
         ref: main
     prompt: |
-      I'm in a checkout of azure-rest-api-specs. Using the release-planner
+      Using the release-planner
       flow, generate SDK for all languages for the Contoso Widget Manager
       release plan. Here is the context you need:
         - release plan work item ID: "29262"
@@ -135,10 +135,6 @@ stimuli:
       max_turns: 8
       max_tokens: 10000
     graders:
-      - type: skill-invocation
-        config:
-          required:
-            - azsdk-common-prepare-release-plan
       - type: tool-calls
         config:
           required:
@@ -156,7 +152,7 @@ stimuli:
         source: C:/Users/gaoh/.vally-cache/azure-rest-api-specs
         ref: main
     prompt: |
-      I'm in a checkout of azure-rest-api-specs. Using the release-planner
+      Using the release-planner
       flow, update the API spec pull request on an existing Contoso Widget
       Manager release plan. Here is the context you need:
         - release plan work item ID: "29262"
@@ -187,7 +183,7 @@ stimuli:
         source: C:/Users/gaoh/.vally-cache/azure-rest-api-specs
         ref: main
     prompt: |
-      I'm in a checkout of azure-rest-api-specs. Using the release-planner
+      Using the release-planner
       flow, refresh the SDK package-name details on an existing Contoso
       Widget Manager release plan from the on-disk TypeSpec emitter
       configuration. Here is the context you need: