ci(skill-eval): pin @microsoft/vally-cli via lockfile (#15752)

* draft: skill-eval 1ES pipeline + pinned vally-cli package (#15747)

Draft plan for migrating .github/workflows/skill-eval.yml onto the 1ES VM template with an authenticated DevOps npm feed.

Placeholders left for the feed URL / service connection — to be filled in after Tuesday sync with Ray. Not wired up; .github/workflows/skill-eval.yml remains the source of truth until the ADO pipeline is registered and green.

* ci(skill-eval): pin vally-cli via lockfile, drop 1ES migration draft

Use `npm ci` against the committed lockfile in `eng/skill-eval/` so
the vally version and its full transitive dependency tree are
reproducible across runs. Drops the global
`npm install -g @microsoft/vally-cli@0.4.0` install; vally is now
invoked directly from `node_modules/.bin`.

Path triggers extended to include `eng/skill-eval/**` and the
workflow file itself so lockfile, package.json, and workflow changes
re-run the lint check.

Deletes the draft `.azure-pipelines/skill-eval.yml` - the 1ES /
DevOps-feed migration is deferred indefinitely; staying on
`ubuntu-latest` + the public npm registry.

Refs #15747

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* docs(skill-eval): correct README/package.json to describe lockfile pinning (not DevOps feed)

Addresses Copilot review feedback on #15752: the workflow uses npm ci against a committed lockfile from the public npm registry; no DevOps feed is involved. Also recommends contributors reproduce CI by running npm ci + local binary instead of a global install.

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Author: helen229

.github/workflows/skill-eval.yml

@@ -5,9 +5,13 @@ on:
     branches: [main]
     paths:
       - '.github/skills/**'
+      - 'eng/skill-eval/**'
+      - '.github/workflows/skill-eval.yml'
   pull_request:
     paths:
       - '.github/skills/**'
+      - 'eng/skill-eval/**'
+      - '.github/workflows/skill-eval.yml'
   workflow_dispatch:
 
 permissions:
@@ -23,9 +27,12 @@ jobs:
       - uses: actions/setup-node@v6
         with:
           node-version: '22'
+          cache: 'npm'
+          cache-dependency-path: eng/skill-eval/package-lock.json
 
-      - name: Install Vally CLI
-        run: npm install -g @microsoft/vally-cli@0.5.0
+      - name: Install Vally CLI (pinned via lockfile)
+        working-directory: eng/skill-eval
+        run: npm ci
 
       - name: Lint skills
-        run: vally lint .
+        run: ./eng/skill-eval/node_modules/.bin/vally lint .

eng/skill-eval/README.md

@@ -0,0 +1,28 @@
+# skill-eval (CI-only npm project)
+
+This folder exists solely to give the [Skill Evaluations GitHub Actions workflow](../../.github/workflows/skill-eval.yml) a tiny `package.json` it can `npm ci` against, so `@microsoft/vally-cli` and its full transitive dependency tree are pinned by the committed `package-lock.json` instead of resolved fresh from semver ranges on every CI run.
+
+- Do not add runtime code here.
+- The only dependency should be `@microsoft/vally-cli`, pinned to the version CI should validate skills with.
+- `package-lock.json` must be committed so `npm ci` is deterministic.
+
+## Updating the Vally CLI version
+
+1. Bump `@microsoft/vally-cli` in `package.json`.
+2. Run `npm install` locally to refresh `package-lock.json`.
+3. Commit both files in the same PR. The workflow's path triggers include `eng/skill-eval/**`, so CI will re-run `vally lint` against the new version automatically.
+
+## Local skill linting
+
+Reproduce exactly what CI does by installing from the same lockfile and invoking the local binary:
+
+```sh
+cd eng/skill-eval
+npm ci
+cd ../..
+./eng/skill-eval/node_modules/.bin/vally lint .
+```
+
+This matches the CI job step-for-step, so a green local run on the current lockfile means a green CI run.
+
+A global install (`npm install -g @microsoft/vally-cli@<version>`) still works for ad-hoc iteration, but it won't match the transitive dependency tree CI uses and isn't a substitute for the steps above when validating a version bump.