diff --git a/demos/anthropic-messages/.gitignore b/demos/anthropic-messages/.gitignore new file mode 100644 index 0000000..8a37f32 --- /dev/null +++ b/demos/anthropic-messages/.gitignore @@ -0,0 +1,3 @@ +# Generated by envsubst from the .tmpl files at demo run time -- see README.md. +passthrough.yaml +transform.yaml diff --git a/demos/anthropic-messages/README.md b/demos/anthropic-messages/README.md index e4c6a91..1f5d83c 100644 --- a/demos/anthropic-messages/README.md +++ b/demos/anthropic-messages/README.md @@ -9,12 +9,20 @@ to different backends with optional format transformation. |------|-------------| | `goals.md` | One-page summary of what this demo covers | | `demo.md` | Step-by-step walkthrough with curl commands | -| `passthrough.yaml` | Config: classify, validate JSON envelopes, route by model to Anthropic API or vLLM | -| `transform.yaml` | Config: transform Anthropic Messages to OpenAI Chat Completions | +| `passthrough.yaml.tmpl` | Config template: classify, validate JSON envelopes, route by model to Anthropic API or vLLM | +| `transform.yaml.tmpl` | Config template: transform Anthropic Messages to OpenAI Chat Completions | + +Both `.tmpl` files are expanded via `envsubst` into `passthrough.yaml` / +`transform.yaml` (git-ignored, regenerated each run) — see Quick Start below. ## Quick Start ```bash +# Point at your vLLM endpoint (defaults to 10.0.0.99:8000 if unset) +export VLLM_ENDPOINT="${VLLM_ENDPOINT:-10.0.0.99:8000}" +envsubst < passthrough.yaml.tmpl > passthrough.yaml +envsubst < transform.yaml.tmpl > transform.yaml + # Passthrough (demos 1-3) export ANTHROPIC_API_KEY=sk-ant-... RUST_LOG=praxis_filter=debug cargo run -p praxis-proxy --release -- \ @@ -28,6 +36,7 @@ RUST_LOG=praxis_filter=debug cargo run -p praxis-proxy --release -- \ ## Prerequisites - **Praxis** built from source (`cargo build -p praxis-proxy --release`) +- `envsubst` (`brew install gettext` on macOS) to render the `.tmpl` configs - `ANTHROPIC_API_KEY` env var set (for Anthropic API passthrough) -- vLLM endpoint at `10.0.0.99:8000` (update `passthrough.yaml` / - `transform.yaml` for your setup) +- A reachable vLLM endpoint — set `VLLM_ENDPOINT` (defaults to `10.0.0.99:8000`, + a placeholder unlikely to be reachable from your network) diff --git a/demos/anthropic-messages/demo.md b/demos/anthropic-messages/demo.md index ca0696f..62f5135 100644 --- a/demos/anthropic-messages/demo.md +++ b/demos/anthropic-messages/demo.md @@ -5,6 +5,12 @@ ## Setup ``` +# Render configs (defaults to a placeholder vLLM endpoint -- set VLLM_ENDPOINT +# to point at a real one) +export VLLM_ENDPOINT="${VLLM_ENDPOINT:-10.0.0.99:8000}" +envsubst < passthrough.yaml.tmpl > passthrough.yaml +envsubst < transform.yaml.tmpl > transform.yaml + # Terminal 1 (Praxis logs) RUST_LOG=praxis_filter=debug cargo run -p praxis-proxy --release -- -c passthrough.yaml 2>&1 \ | grep -E 'classified|validation|route matched|upstream selected|credential|transformed|streaming' diff --git a/demos/anthropic-messages/passthrough.yaml b/demos/anthropic-messages/passthrough.yaml.tmpl similarity index 96% rename from demos/anthropic-messages/passthrough.yaml rename to demos/anthropic-messages/passthrough.yaml.tmpl index 4f0d859..89af468 100644 --- a/demos/anthropic-messages/passthrough.yaml +++ b/demos/anthropic-messages/passthrough.yaml.tmpl @@ -39,4 +39,4 @@ filter_chains: sni: "api.anthropic.com" - name: vllm endpoints: - - "10.0.0.99:8000" + - "${VLLM_ENDPOINT}" diff --git a/demos/anthropic-messages/transform.yaml b/demos/anthropic-messages/transform.yaml.tmpl similarity index 96% rename from demos/anthropic-messages/transform.yaml rename to demos/anthropic-messages/transform.yaml.tmpl index a499b42..98852f8 100644 --- a/demos/anthropic-messages/transform.yaml +++ b/demos/anthropic-messages/transform.yaml.tmpl @@ -36,4 +36,4 @@ filter_chains: clusters: - name: vllm endpoints: - - "10.0.0.99:8000" + - "${VLLM_ENDPOINT}" diff --git a/demos/maas-ipp/README.md b/demos/maas-ipp/README.md index d50b74b..f4f413e 100644 --- a/demos/maas-ipp/README.md +++ b/demos/maas-ipp/README.md @@ -10,6 +10,37 @@ Forge owns cluster lifecycle and infra stacks. `maas-controller` **owns EnvoyFil This profile is for MaaS + Praxis integration work. It is **not** the Grid multi-cluster GLB demo, and it intentionally diverges from issue #2’s “CRDs-only / skip Authorino” simulation table. +## What this demo proves + +MaaS (Models-as-a-Service on OpenShift AI) already ships a payload-processing +step — **IPP** — as an Envoy `ExternalProcessor` in front of every model call, +split into an **IPP-pre** hop (before Kuadrant auth: extract model identity, +resolve the MaaS auth identity, apply bounded request mutations) and an +**IPP-post** hop (after auth: protect the provider boundary, select the +provider, translate protocol, inject provider credentials). Today that +processor is a stock ODH container (`odh-ai-gateway-payload-processing`). + +This demo swaps that one container for **Praxis**, with nothing else in the +stock MaaS datapath changed — same Gateway, same Kuadrant `AuthPolicy`, same +CRDs, same HTTPRoute. `MAAS_IPP_PROFILE` is the switch: + +| `MAAS_IPP_PROFILE` | IPP-pre/IPP-post implementation | Platform manifests overlay | +|---|---|---| +| `llm-d` (stock) | `odh-ai-gateway-payload-processing` (`$IPP_IMAGE`) | `overlays/xks` | +| `praxis` (this lab's default) | Praxis (`$PRAXIS_EXTPROC_IMAGE`), implementing the same Pre-Auth/Post-Auth contract | `overlays/xks-praxis` | + +That table is the actual point of the demo: it's evidence Praxis can sit in +MaaS's existing `ExternalProcessor` slot and preserve IPP's pre/post-auth +behavior, not a new, separate datapath that MaaS would have to adopt +wholesale. From a roadmap perspective, that's the difference between "replace +your gateway" and "replace one container" — the latter is a far smaller ask +of a platform team already running MaaS in production. + +The `## Call models` walkthrough below exercises both hops end-to-end +(model-identity extraction pre-auth, provider selection and credential +injection post-auth) against two backend shapes MaaS supports: an in-cluster +`LLMInferenceService` sim and an `ExternalModel`. + ## Pins Version and namespace pins live in `forge.yaml` cluster `properties`. Stacks @@ -107,6 +138,13 @@ echo "$API_KEY" ## Call models +> Every request below crosses the full stock datapath — `Client → Istio +> Gateway → IPP-pre → Kuadrant Auth → IPP-post → HTTPRoute → LLM sim` — with +> Praxis standing in for IPP-pre/IPP-post per `MAAS_IPP_PROFILE=praxis`. What +> to watch: the request succeeds identically to how it would against the +> stock `llm-d` IPP profile — that equivalence, not the model's answer, is +> the thing this demo is proving. + Gateway LB (MetalLB on the Kind docker network — reachable from the Kind host): ```bash @@ -116,6 +154,14 @@ GW=$(kubectl --context kind-maas-ipp-local -n istio-system \ **Internal llm-d sim** (`LLMInferenceService` `sim-internal`): +> Praxis's IPP-pre hop reads the `/llm-internal/sim-internal/...` path and +> `model` field to resolve which MaaS-registered model this call targets +> *before* Kuadrant decides whether the bearer token is allowed to call it. +> Post-auth, IPP-post resolves `sim-internal` to its backing +> `LLMInferenceService` and forwards — the same provider-selection step the +> stock IPP container performs, just implemented by Praxis's filter chain +> instead. + ```bash curl -sk "https://${GW}/llm-internal/sim-internal/v1/chat/completions" \ -H "Authorization: Bearer ${API_KEY}" \ @@ -125,6 +171,11 @@ curl -sk "https://${GW}/llm-internal/sim-internal/v1/chat/completions" \ **External model** (`ExternalModel` `llm-katan-openai` — remote simulator must be reachable): +> Same pre/post-auth contract, but IPP-post now resolves to an `ExternalModel` +> instead of an in-cluster `LLMInferenceService` — the provider-boundary and +> credential-injection step that matters most for customers routing to +> models MaaS itself doesn't host. + ```bash curl -sk "https://${GW}/llm/llm-katan-openai/v1/chat/completions" \ -H "Authorization: Bearer ${API_KEY}" \