Validate LiteLLM config candidates

Author: RamazanKara

CHANGELOG.md

@@ -30,6 +30,7 @@
 - Added Forgejo migration candidate validation for the GitHub Enterprise assessment path.
 - Added identity realm/client candidate validation for the Okta/Auth0 assessment path.
 - Added edge VCL, HAProxy, and Coraza candidate validation for the Cloudflare/Akamai assessment path.
+- Added LiteLLM/vLLM config candidate validation for the OpenAI/Anthropic assessment path.
 - Updated CI and release workflows to Node.js 24-native GitHub Actions.
 
 ## 0.1.0 - 2026-05-24

README.md

@@ -116,7 +116,7 @@ Included in the current implementation:
 - OpenTelemetry Collector sketch.
 - ArgoCD starter manifest.
 - Typed migration plan manifest and phase-gate Markdown plan.
-- Validation report with embedded JSON Schema checks, Grafana dashboard, Prometheus alert, OpenTelemetry collector, ArgoCD, Forgejo migration, identity realm/client, and edge VCL/HAProxy/Coraza candidate checks, YAML/JSON parsing, evidence ref checks, secret scan, and optional `promtool`/`kubeconform` checks.
+- Validation report with embedded JSON Schema checks, Grafana dashboard, Prometheus alert, OpenTelemetry collector, ArgoCD, Forgejo migration, identity realm/client, edge VCL/HAProxy/Coraza, and LiteLLM/vLLM candidate checks, YAML/JSON parsing, evidence ref checks, secret scan, and optional `promtool`/`kubeconform` checks.
 - Evidence bundle export.
 - Evidence bundle path-safety checks that reject symlinks in exported project sections.
 - No-op assist provider and explicit opt-in LiteLLM assist.
@@ -164,7 +164,7 @@ Okta/Auth0 to Keycloak/Zitadel collects applications, SAML/OIDC client metadata,
 
 Cloudflare/Akamai to Varnish/HAProxy/Coraza collects DNS records, WAF rules, cache rules, redirects, origins, TLS settings, bot rules, and page rules from live Cloudflare/Akamai APIs or local fixtures. The Akamai collector uses read-only EdgeGrid-authenticated calls for Edge DNS, Property Manager, and optional AppSec metadata. It generates validated VCL, HAProxy, Coraza, cache parity, and WAF enforcement review artifacts.
 
-OpenAI/Anthropic to vLLM/LiteLLM collects model usage classes, token volumes, latency expectations, sensitive prompt categories, tool usage, and fallback behavior from local fixtures. It can also collect model-grouped aggregate OpenAI completions usage, aggregate Anthropic Messages API usage, available model metadata where exposed, server web-search tool metadata, and hourly peak estimates from live provider APIs without storing prompts or credentials. The path generates self-hosted LLM readiness, LiteLLM routing, vLLM sizing, evaluation, and data sensitivity artifacts.
+OpenAI/Anthropic to vLLM/LiteLLM collects model usage classes, token volumes, latency expectations, sensitive prompt categories, tool usage, and fallback behavior from local fixtures. It can also collect model-grouped aggregate OpenAI completions usage, aggregate Anthropic Messages API usage, available model metadata where exposed, server web-search tool metadata, and hourly peak estimates from live provider APIs without storing prompts or credentials. The path generates self-hosted LLM readiness, validated LiteLLM routing, vLLM sizing, evaluation, and data sensitivity artifacts.
 
 ## License
 

docs/architecture.md

@@ -14,6 +14,6 @@ Okta/Auth0 to Keycloak/Zitadel reuses the same normalized inventory and evidence
 
 Cloudflare/Akamai to Varnish/HAProxy/Coraza generates and validates edge-specific VCL, HAProxy, and Coraza candidate files plus cache and WAF review reports. It supports fixture import plus read-only live Cloudflare and Akamai collectors.
 
-OpenAI/Anthropic to vLLM/LiteLLM generates AI provider readiness, LiteLLM routing, vLLM sizing, evaluation, and data sensitivity artifacts. It supports fixture import plus read-only live OpenAI/Anthropic aggregate usage collectors.
+OpenAI/Anthropic to vLLM/LiteLLM generates AI provider readiness, validated LiteLLM routing, vLLM sizing, evaluation, and data sensitivity artifacts. It supports fixture import plus read-only live OpenAI/Anthropic aggregate usage collectors.
 
 AI assist is optional and never required for validation or export.

docs/cli.md

@@ -27,11 +27,11 @@ The CI test suite also runs this definition-of-done pipeline against the fixture
 
 The checked-in Datadog example can be refreshed with `make example VERSION=0.1.0-dev`. CI also runs `make example-smoke` as part of `make verify` to ensure the example fixture still completes the full pipeline.
 
-`openexit validate` performs typed consistency checks, embedded JSON Schema validation, Grafana dashboard candidate validation, Prometheus alert-rule candidate validation, OpenTelemetry collector candidate validation, ArgoCD candidate validation, Forgejo migration candidate validation, identity realm/client candidate validation, edge VCL/HAProxy/Coraza candidate validation, YAML/JSON parse checks, evidence reference checks, secret scanning, and optional external tool checks when `promtool` or `kubeconform` are installed.
+`openexit validate` performs typed consistency checks, embedded JSON Schema validation, Grafana dashboard candidate validation, Prometheus alert-rule candidate validation, OpenTelemetry collector candidate validation, ArgoCD candidate validation, Forgejo migration candidate validation, identity realm/client candidate validation, edge VCL/HAProxy/Coraza candidate validation, LiteLLM/vLLM candidate validation, YAML/JSON parse checks, evidence reference checks, secret scanning, and optional external tool checks when `promtool` or `kubeconform` are installed.
 
 `openexit export` refuses to package symlinks from exported project sections, even with `--force`, so evidence bundles cannot accidentally include files from outside the project tree.
 
-Generate individual artifacts with `openexit generate --artifact <name>`. The primary Datadog path supports `mapping`, `assessment`, `risk-register`, `manual-review`, `cost-drivers`, `target-architecture`, `acceptance-criteria`, `rollback-plan`, `runbook`, `restore-drill-checklist`, `alert-shadowing-plan`, `migration-plan`, `grafana-dashboards`, `prometheus-rules`, `opentelemetry`, and `argocd`. The GitHub Enterprise path also supports `forgejo-migration-candidate`; the identity path also supports `realm-client-candidate`; the edge path also supports `vcl-candidates`, `haproxy-candidates`, and `coraza-rule-candidates`.
+Generate individual artifacts with `openexit generate --artifact <name>`. The primary Datadog path supports `mapping`, `assessment`, `risk-register`, `manual-review`, `cost-drivers`, `target-architecture`, `acceptance-criteria`, `rollback-plan`, `runbook`, `restore-drill-checklist`, `alert-shadowing-plan`, `migration-plan`, `grafana-dashboards`, `prometheus-rules`, `opentelemetry`, and `argocd`. The GitHub Enterprise path also supports `forgejo-migration-candidate`; the identity path also supports `realm-client-candidate`; the edge path also supports `vcl-candidates`, `haproxy-candidates`, and `coraza-rule-candidates`; the AI provider path also supports `litellm-config-candidate`.
 
 The GitHub Enterprise to Forgejo fixture path uses local JSON metadata:
 

docs/openai-anthropic-to-vllm-litellm.md

@@ -106,4 +106,6 @@ openexit validate --project ./anthropic-live
 - `assessment/data-sensitivity-report.md`
 - `generated-config/ai/litellm/config.candidate.yaml`
 
+Validation checks the LiteLLM candidate for source/target metadata, non-production safety flags, absence of provider credentials, explicit LiteLLM/vLLM router policy, one route per captured usage class, preserved source models, owners, latency and fallback metadata, runtime credential placeholders, vLLM sizing assumptions, and evidence refs.
+
 All generated target files are candidates and require human review, model evaluation, load testing, data-control review, and tool-policy review before production use.

docs/release.md

@@ -16,7 +16,7 @@ Release-blocking v0.1 requirements:
 - Markdown handover artifacts.
 - Grafana, Prometheus, OpenTelemetry, and ArgoCD candidate generators.
 - Typed migration plan manifest with assessment, pilot, shadow, and cutover phase gates.
-- Validation engine with embedded JSON Schema checks, Grafana dashboard, Prometheus alert, OpenTelemetry collector, ArgoCD, Forgejo migration, identity realm/client, and edge VCL/HAProxy/Coraza candidate checks, YAML/JSON parsing, evidence refs, secret scan, optional promtool, and optional kubeconform.
+- Validation engine with embedded JSON Schema checks, Grafana dashboard, Prometheus alert, OpenTelemetry collector, ArgoCD, Forgejo migration, identity realm/client, edge VCL/HAProxy/Coraza, and LiteLLM/vLLM candidate checks, YAML/JSON parsing, evidence refs, secret scan, optional promtool, and optional kubeconform.
 - Evidence bundle export with checksums and OpenExit version metadata.
 - No-op AI assist and optional external assist behind explicit opt-in.
 - Documentation, examples, CI, release draft workflow, and reproducible release artifacts.
@@ -54,6 +54,7 @@ The AI provider path is complete for local fixture assessment workflows and incl
 - [ ] `validation/validation-report.md` includes `forgejo-migration-candidate: passed` for the GitHub Enterprise to Forgejo pipeline.
 - [ ] `validation/validation-report.md` includes `identity-realm-client-candidate: passed` for the Okta/Auth0 to Keycloak/Zitadel pipeline.
 - [ ] `validation/validation-report.md` includes `edge-candidates: passed` for the Cloudflare/Akamai to Varnish/HAProxy/Coraza pipeline.
+- [ ] `validation/validation-report.md` includes `litellm-config-candidate: passed` for the OpenAI/Anthropic to vLLM/LiteLLM pipeline.
 - [ ] `CHANGELOG.md` has a `0.1.0` section.
 - [ ] Exported bundle README includes version, commit, build date, bundle timestamp, and candidate warning.
 - [ ] Export refuses symlinks in exported project sections, including when `--force` is used.

internal/app/e2e_test.go

@@ -720,20 +720,7 @@ func TestValidationRejectsBrokenEdgeHAProxyCandidate(t *testing.T) {
 
 func TestAIProviderFixturePipeline(t *testing.T) {
 	projectDir := filepath.Join(t.TempDir(), "ai-demo")
-	fixturePath := filepath.Join("..", "..", "testdata", "ai-provider", "small.json")
-
-	commands := [][]string{
-		{"init", projectDir, "--source", "ai-provider", "--target", "vllm-litellm"},
-		{"collect", "ai-fixture", "--project", projectDir, "--input", fixturePath},
-		{"assess", "--project", projectDir, "--target", "vllm-litellm"},
-		{"generate", "--project", projectDir, "--all"},
-		{"validate", "--project", projectDir},
-	}
-	for _, args := range commands {
-		if err := executeForTest(args...); err != nil {
-			t.Fatalf("openexit %s failed: %v", strings.Join(args, " "), err)
-		}
-	}
+	runAIProviderFixturePipeline(t, projectDir)
 	for _, rel := range []string{
 		"inventory/openexit.inventory.yaml",
 		"mapping/openexit.mapping.yaml",
@@ -770,6 +757,67 @@ func TestAIProviderFixturePipeline(t *testing.T) {
 			t.Fatalf("expected assessment finding %s", id)
 		}
 	}
+	report, err := os.ReadFile(filepath.Join(projectDir, "validation", "validation-report.md"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !strings.Contains(string(report), "litellm-config-candidate: passed") {
+		t.Fatalf("expected LiteLLM candidate validation marker, got:\n%s", string(report))
+	}
+}
+
+func TestValidationRejectsProductionReadyLiteLLMCandidate(t *testing.T) {
+	projectDir := filepath.Join(t.TempDir(), "ai-demo")
+	runAIProviderFixturePipeline(t, projectDir)
+	candidatePath := filepath.Join(projectDir, "generated-config", "ai", "litellm", "config.candidate.yaml")
+	data, err := os.ReadFile(candidatePath)
+	if err != nil {
+		t.Fatal(err)
+	}
+	corrupt := strings.Replace(string(data), "productionReady: false", "productionReady: true", 1)
+	if corrupt == string(data) {
+		t.Fatal("test fixture did not contain productionReady marker")
+	}
+	if err := os.WriteFile(candidatePath, []byte(corrupt), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	if err := executeForTest("validate", "--project", projectDir); err == nil {
+		t.Fatal("expected validation to fail when LiteLLM candidate is marked production ready")
+	}
+	report, err := os.ReadFile(filepath.Join(projectDir, "validation", "validation-report.md"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !strings.Contains(string(report), "litellm-config-candidate") || !strings.Contains(string(report), "productionReady") {
+		t.Fatalf("expected LiteLLM candidate productionReady failure in validation report, got:\n%s", string(report))
+	}
+}
+
+func TestValidationRejectsBrokenLiteLLMRouteCandidate(t *testing.T) {
+	projectDir := filepath.Join(t.TempDir(), "ai-demo")
+	runAIProviderFixturePipeline(t, projectDir)
+	candidatePath := filepath.Join(projectDir, "generated-config", "ai", "litellm", "config.candidate.yaml")
+	data, err := os.ReadFile(candidatePath)
+	if err != nil {
+		t.Fatal(err)
+	}
+	corrupt := strings.Replace(string(data), "model: vllm/support-chat-replacement-for-gpt-4o", "model: vllm/support-chat-unreviewed", 1)
+	if corrupt == string(data) {
+		t.Fatal("test fixture did not contain LiteLLM model route")
+	}
+	if err := os.WriteFile(candidatePath, []byte(corrupt), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	if err := executeForTest("validate", "--project", projectDir); err == nil {
+		t.Fatal("expected validation to fail when LiteLLM route drifts from inventory")
+	}
+	report, err := os.ReadFile(filepath.Join(projectDir, "validation", "validation-report.md"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !strings.Contains(string(report), "litellm-config-candidate") || !strings.Contains(string(report), "litellm_params") {
+		t.Fatalf("expected LiteLLM route failure in validation report, got:\n%s", string(report))
+	}
 }
 
 func TestExportRefusesInvalidProjectWithoutForce(t *testing.T) {
@@ -838,6 +886,23 @@ func runEdgeFixturePipeline(t *testing.T, projectDir string) {
 	}
 }
 
+func runAIProviderFixturePipeline(t *testing.T, projectDir string) {
+	t.Helper()
+	fixturePath := filepath.Join("..", "..", "testdata", "ai-provider", "small.json")
+	commands := [][]string{
+		{"init", projectDir, "--source", "ai-provider", "--target", "vllm-litellm"},
+		{"collect", "ai-fixture", "--project", projectDir, "--input", fixturePath},
+		{"assess", "--project", projectDir, "--target", "vllm-litellm"},
+		{"generate", "--project", projectDir, "--all"},
+		{"validate", "--project", projectDir},
+	}
+	for _, args := range commands {
+		if err := executeForTest(args...); err != nil {
+			t.Fatalf("openexit %s failed: %v", strings.Join(args, " "), err)
+		}
+	}
+}
+
 func executeForTest(args ...string) error {
 	cmd := NewRootCommand()
 	cmd.SetArgs(args)