feat(phase-56): Honest Benchmarks — real recall@k, locomo10.json, HOLD comparison marketing #55
RichardHightowere2e-cli.yml
on: pull_request
Matrix: e2e-cli
CLI Matrix Report
8s
Annotations
21 warnings
|
E2E CLI - gemini (ubuntu-24.04)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4, actions/upload-artifact@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
E2E CLI - copilot (ubuntu-24.04)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4, actions/upload-artifact@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
E2E CLI - codex (ubuntu-24.04)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4, actions/upload-artifact@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
E2E CLI - claude-code (ubuntu-24.04)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4, actions/upload-artifact@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
E2E CLI - opencode (ubuntu-24.04)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4, actions/upload-artifact@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
E2E CLI - gemini (macos-latest)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4, actions/upload-artifact@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
E2E CLI - gemini (macos-latest)
The following taps are not trusted:
aws/tap
Homebrew is currently ignoring formulae, casks and commands from these taps because tap trust is required.
Untap them with:
brew untap aws/tap
Trust specific formulae, casks and commands with:
brew trust --formula <user>/<tap>/<formula>
brew trust --cask <user>/<tap>/<cask>
brew trust --command <user>/<tap>/<command>
Whole-tap trust is broader and includes all current and future formulae,
casks and commands from the listed taps. Trust whole taps with:
brew trust aws/tap
To disable trust checks:
export HOMEBREW_NO_REQUIRE_TAP_TRUST=1
This is not recommended and will be removed in a later release.
For more information, see:
https://docs.brew.sh/Tap-Trust
|
|
E2E CLI - gemini (macos-latest)
The following taps are not trusted:
aws/tap
Homebrew is currently ignoring formulae, casks and commands from these taps because tap trust is required.
Untap them with:
brew untap aws/tap
Trust specific formulae, casks and commands with:
brew trust --formula <user>/<tap>/<formula>
brew trust --cask <user>/<tap>/<cask>
brew trust --command <user>/<tap>/<command>
Whole-tap trust is broader and includes all current and future formulae,
casks and commands from the listed taps. Trust whole taps with:
brew trust aws/tap
To disable trust checks:
export HOMEBREW_NO_REQUIRE_TAP_TRUST=1
This is not recommended and will be removed in a later release.
For more information, see:
https://docs.brew.sh/Tap-Trust
|
|
E2E CLI - codex (macos-latest)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4, actions/upload-artifact@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
E2E CLI - codex (macos-latest)
The following taps are not trusted:
aws/tap
Homebrew is currently ignoring formulae, casks and commands from these taps because tap trust is required.
Untap them with:
brew untap aws/tap
Trust specific formulae, casks and commands with:
brew trust --formula <user>/<tap>/<formula>
brew trust --cask <user>/<tap>/<cask>
brew trust --command <user>/<tap>/<command>
Whole-tap trust is broader and includes all current and future formulae,
casks and commands from the listed taps. Trust whole taps with:
brew trust aws/tap
To disable trust checks:
export HOMEBREW_NO_REQUIRE_TAP_TRUST=1
This is not recommended and will be removed in a later release.
For more information, see:
https://docs.brew.sh/Tap-Trust
|
|
E2E CLI - codex (macos-latest)
The following taps are not trusted:
aws/tap
Homebrew is currently ignoring formulae, casks and commands from these taps because tap trust is required.
Untap them with:
brew untap aws/tap
Trust specific formulae, casks and commands with:
brew trust --formula <user>/<tap>/<formula>
brew trust --cask <user>/<tap>/<cask>
brew trust --command <user>/<tap>/<command>
Whole-tap trust is broader and includes all current and future formulae,
casks and commands from the listed taps. Trust whole taps with:
brew trust aws/tap
To disable trust checks:
export HOMEBREW_NO_REQUIRE_TAP_TRUST=1
This is not recommended and will be removed in a later release.
For more information, see:
https://docs.brew.sh/Tap-Trust
|
|
E2E CLI - opencode (macos-latest)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4, actions/upload-artifact@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
E2E CLI - opencode (macos-latest)
The following taps are not trusted:
aws/tap
Homebrew is currently ignoring formulae, casks and commands from these taps because tap trust is required.
Untap them with:
brew untap aws/tap
Trust specific formulae, casks and commands with:
brew trust --formula <user>/<tap>/<formula>
brew trust --cask <user>/<tap>/<cask>
brew trust --command <user>/<tap>/<command>
Whole-tap trust is broader and includes all current and future formulae,
casks and commands from the listed taps. Trust whole taps with:
brew trust aws/tap
To disable trust checks:
export HOMEBREW_NO_REQUIRE_TAP_TRUST=1
This is not recommended and will be removed in a later release.
For more information, see:
https://docs.brew.sh/Tap-Trust
|
|
E2E CLI - opencode (macos-latest)
The following taps are not trusted:
aws/tap
Homebrew is currently ignoring formulae, casks and commands from these taps because tap trust is required.
Untap them with:
brew untap aws/tap
Trust specific formulae, casks and commands with:
brew trust --formula <user>/<tap>/<formula>
brew trust --cask <user>/<tap>/<cask>
brew trust --command <user>/<tap>/<command>
Whole-tap trust is broader and includes all current and future formulae,
casks and commands from the listed taps. Trust whole taps with:
brew trust aws/tap
To disable trust checks:
export HOMEBREW_NO_REQUIRE_TAP_TRUST=1
This is not recommended and will be removed in a later release.
For more information, see:
https://docs.brew.sh/Tap-Trust
|
|
E2E CLI - claude-code (macos-latest)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4, actions/upload-artifact@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
E2E CLI - claude-code (macos-latest)
The following taps are not trusted:
aws/tap
Homebrew is currently ignoring formulae, casks and commands from these taps because tap trust is required.
Untap them with:
brew untap aws/tap
Trust specific formulae, casks and commands with:
brew trust --formula <user>/<tap>/<formula>
brew trust --cask <user>/<tap>/<cask>
brew trust --command <user>/<tap>/<command>
Whole-tap trust is broader and includes all current and future formulae,
casks and commands from the listed taps. Trust whole taps with:
brew trust aws/tap
To disable trust checks:
export HOMEBREW_NO_REQUIRE_TAP_TRUST=1
This is not recommended and will be removed in a later release.
For more information, see:
https://docs.brew.sh/Tap-Trust
|
|
E2E CLI - claude-code (macos-latest)
The following taps are not trusted:
aws/tap
Homebrew is currently ignoring formulae, casks and commands from these taps because tap trust is required.
Untap them with:
brew untap aws/tap
Trust specific formulae, casks and commands with:
brew trust --formula <user>/<tap>/<formula>
brew trust --cask <user>/<tap>/<cask>
brew trust --command <user>/<tap>/<command>
Whole-tap trust is broader and includes all current and future formulae,
casks and commands from the listed taps. Trust whole taps with:
brew trust aws/tap
To disable trust checks:
export HOMEBREW_NO_REQUIRE_TAP_TRUST=1
This is not recommended and will be removed in a later release.
For more information, see:
https://docs.brew.sh/Tap-Trust
|
|
E2E CLI - copilot (macos-latest)
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4, actions/upload-artifact@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
E2E CLI - copilot (macos-latest)
The following taps are not trusted:
aws/tap
Homebrew is currently ignoring formulae, casks and commands from these taps because tap trust is required.
Untap them with:
brew untap aws/tap
Trust specific formulae, casks and commands with:
brew trust --formula <user>/<tap>/<formula>
brew trust --cask <user>/<tap>/<cask>
brew trust --command <user>/<tap>/<command>
Whole-tap trust is broader and includes all current and future formulae,
casks and commands from the listed taps. Trust whole taps with:
brew trust aws/tap
To disable trust checks:
export HOMEBREW_NO_REQUIRE_TAP_TRUST=1
This is not recommended and will be removed in a later release.
For more information, see:
https://docs.brew.sh/Tap-Trust
|
|
E2E CLI - copilot (macos-latest)
The following taps are not trusted:
aws/tap
Homebrew is currently ignoring formulae, casks and commands from these taps because tap trust is required.
Untap them with:
brew untap aws/tap
Trust specific formulae, casks and commands with:
brew trust --formula <user>/<tap>/<formula>
brew trust --cask <user>/<tap>/<cask>
brew trust --command <user>/<tap>/<command>
Whole-tap trust is broader and includes all current and future formulae,
casks and commands from the listed taps. Trust whole taps with:
brew trust aws/tap
To disable trust checks:
export HOMEBREW_NO_REQUIRE_TAP_TRUST=1
This is not recommended and will be removed in a later release.
For more information, see:
https://docs.brew.sh/Tap-Trust
|
|
CLI Matrix Report
Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@v4, actions/download-artifact@v4, actions/upload-artifact@v4. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
Artifacts
Produced during runtime
| Name | Size | Digest | |
|---|---|---|---|
|
cli-matrix-report
|
12.1 KB |
sha256:dc96c2d3e97a3dcb48905583459e7c1d8fc10d996408dd243d073768c79c37f0
|
|
|
junit-claude-code-macos-latest
|
1.19 KB |
sha256:207661829391f633e1cde20ca2a609c2ca12b5a1d2775b0b448e6dd5e020a797
|
|
|
junit-claude-code-ubuntu-24.04
|
1.08 KB |
sha256:5a89e499cf1f2eb5233406d9f1bf0ce5b3d3e0f064654b7fd11980ea8e973456
|
|
|
junit-codex-macos-latest
|
1.13 KB |
sha256:5132211534dfff5e7380c5a092f87f2b27a36585b5549f9e6d47324b9e735083
|
|
|
junit-codex-ubuntu-24.04
|
1.08 KB |
sha256:1f328a538893c4f1aeba844ea591f0bae37cfc88a55aabfb58d83d71f83cf590
|
|
|
junit-copilot-macos-latest
|
1.18 KB |
sha256:217186de9159cb7d5b03b4b9218ad395b0239fe8d0c6ca80efe25ecc352c8b1d
|
|
|
junit-copilot-ubuntu-24.04
|
1.13 KB |
sha256:86a640acf2d32c1dae17c39de2cc61b6fb7eedaed0b19ba4809bf0fa3f63b02d
|
|
|
junit-gemini-macos-latest
|
1.35 KB |
sha256:77461347d4bf441bec9a56480a614b61a0c4d7b66997a3a5d8ae35455cbfa09f
|
|
|
junit-gemini-ubuntu-24.04
|
1.28 KB |
sha256:d5908d21af14582f9364bfab84f2d92ea08f5b4323b3386ff8a83cbf91d2983a
|
|
|
junit-opencode-macos-latest
|
1.21 KB |
sha256:83109ff69c2b167caf022d000e5ab707dfb6d91c58c9a1a91bb2f8d11d9d8c15
|
|
|
junit-opencode-ubuntu-24.04
|
1.13 KB |
sha256:1336505673dc1fefe111249e0b0a2bf84f22c6526b5798aed0af835b9f389da7
|
|