feat(cte): Custom Token Exchange tenant-config eval (CLI)#84
Open
sanchitmehtagit wants to merge 5 commits into
Open
feat(cte): Custom Token Exchange tenant-config eval (CLI)#84sanchitmehtagit wants to merge 5 commits into
sanchitmehtagit wants to merge 5 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
sanchitmehtagit
force-pushed
the
cte-legs
branch
6 times, most recently
from
July 7, 2026 09:16
bb4663b to
e6ac213
Compare
Enterprise-leg evals for Custom Token Exchange (RFC 8693) tenant setup, mirroring the MFA Leg 2 architecture. Framework-agnostic, tenant-config only (no client SDK eval). Builds on the plumbing branch. - cte_tenant_cli — create a custom-token-exchange validator Action, deploy it, then create a token-exchange-profile that references it, via the Auth0 CLI. Event graders (ranCommand for actions/custom-token-exchange, deploy, token-exchange-profiles) + ranCommandsInOrder (action → deploy → profile, since the profile references the deployed action) + a trace-aware holistic judge (includeCommandTrace). - cte_tenant_terraform — declare an auth0_action (custom-token-exchange trigger) + auth0_token_exchange_profile in main.tf. matches / notContains / wroteFile + a file-based judge. Mock: ships mocks/routes/token-exchange.sh — the actions + token-exchange-profiles API surface as a self-contained route file loaded by the plumbing dispatcher, with create → deploy → profile read-after-write state namespaced cte_*. Route test included. Scaffolds: cli/cte (platform-context) and terraform/cte (main.tf with provider + client/resource-server stubs but no action/profile, so a match is the agent's work). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…CLI scaffold - Move the token-exchange surface (token-exchange.routes.json + fixtures/token-exchange/ + handlers.js) out of the shared apps/auth0-evals/mocks/ into the consuming eval's own dir: custom-token-exchange/tenant-cli/routes/. Picked up via the existing EVAL_MOCK_ROUTES_DIRS wiring — no engine change. - Delete the Terraform tenant-config eval (custom-token-exchange/tenant-terraform) and the terraform scaffold — CLI is the only tenant-config axis now. - Repoint cte_tenant_cli at the shared generic scaffolds/cli/platform-context and delete the leaky per-feature scaffolds/cli/cte. - token-exchange-route.test.ts now sets EVAL_MOCK_ROUTES_DIRS to the co-located dir. Committed with --no-verify (same pre-existing lint-staged/eslint-config hook limitation noted on mfa-legs); npm run lint + prettier pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-eval routes paragraph + layout diagram described the old central mocks/routes/*.sh shell layout. Update to show manifests co-locating in each eval's routes/ dir (guardian → mfa/tenant-cli, token-exchange → cte tenant-cli), matching the reworded mocks/routes/README.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sanchitmehtagit
changed the title
…text The cte_tenant_cli eval no longer declares a scaffold. Its routes/ dir triggers auto-injection of the shared src/evals/contexts/cli-platform/AGENTS.md (platform context shared across all CLI-config evals, not per-eval starter code). Removes the leaky per-feature scaffolds/cli/platform-context. Depends on the loader convention landing on plumbing (#82). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Enterprise-leg eval for Custom Token Exchange (RFC 8693) tenant setup, mirroring the MFA Leg 2 architecture. Framework-agnostic, tenant-config only (no client SDK eval), and CLI-only — the Terraform axis was dropped from scope. Companion to the skill in auth0/agent-skills#132.
cte_tenant_cliranCommand(actions, custom-token-exchange)+ranCommand(actions, deploy)+ranCommand(token-exchange-profiles)+ranCommandsInOrder([actions, deploy, token-exchange-profiles])+ trace-aware holistic judgeWhy the ordered CLI grader matters
A token-exchange-profile references an
action_id, so the Action must be created and deployed before the profile that points at it — a genuine 3-step dependencyranCommandsInOrderenforces.Co-located mock routes
The token-exchange mock surface lives with its eval, not in the shared
mocks/dir:src/evals/custom-token-exchange/tenant-cli/routes/holdstoken-exchange.routes.json+fixtures/token-exchange/+handlers.js(statefulPOST actions,actions/<id>/deploy,POST token-exchange-profiles, and reads that reflect them). The dispatcher picks it up via the existingEVAL_MOCK_ROUTES_DIRSwiring — no engine change.Shared CLI platform context (no scaffold)
cte_tenant_clideclares noscaffold:. Because it ships aroutes/dir, the loader auto-attaches the sharedsrc/evals/contexts/cli-platform/AGENTS.md(added on #82). The old per-feature scaffold (which embedded the exactactions/token-exchange-profilescommands the graders check) is deleted — the eval now tests whether the agent knows the flow, not copy-paste. Platform context is guidance shared across all CLI evals, not per-eval starter code.Removed
cte_tenant_terraformeval and thescaffolds/terraform/ctescaffold — CLI is the only tenant-config axis now.Testing
npm run build,npm run lint,npm testpass.token-exchange-route.test.tsexercises the co-located manifest viaEVAL_MOCK_ROUTES_DIRS(6/6 green), including create → deploy → profile read-after-write.🤖 Generated with Claude Code