TIP layer repaired + autonomous learning loop closed end-to-end - #2
Open
renefichtmueller wants to merge 5 commits into
Open
TIP layer repaired + autonomous learning loop closed end-to-end#2renefichtmueller wants to merge 5 commits into
renefichtmueller wants to merge 5 commits into
Conversation
The TIP layer (ADR-0005) referenced a client API that no longer existed:
three adapter packages and the integration test imported createTIPClient /
TIPClientConfig / createEOPulseClient from @llm-gateway/client, none of
which were exported. Every consumer failed to compile, four package
tsconfigs extended a root tsconfig.json that was missing, and CI only ever
type-checked the gateway, so the breakage stayed invisible.
Client SDK (@llm-gateway/client):
- Implement the TIP client (packages/client/src/tip.ts): TIPClient,
TIPClientConfig, createTIPClient with both the ADR-0005 config-object
signature and the legacy URL-string signature. Prompt-oriented
completion over /v1/completion with transparent Ollama fallback
(fallback flag per response), confidence normalized to 0-1,
getStatus() snapshot, and an active health() probe.
- Split the existing task-oriented client into core.ts, re-export both
from index.ts; fix the exactOptionalPropertyTypes violations in the
factory functions and the stale @adaptive-llm-gateway/client header.
- Publish built output (dist + declarations) instead of raw TS source so
adapters compile against declarations and node can run built CLIs.
- Add hermetic unit tests (mocked fetch) for mapping, fallback, both
factory signatures, and health states.
Adapters:
- claude-code-bridge: use the real TIPClient API, delegate health() to
the client, drop the bogus "anthropic" dependency, hermetic tests.
- chatgpt-api-adapter: correct OpenAI chunk type (chat.completion.chunk),
stable stream id per response, whitespace-preserving chunking instead
of char-by-char SSE, reply.hijack() for raw SSE under Fastify v5,
@fastify/cors ^10 to match Fastify v5, honor AGENT_ID, default port
8788, log errors instead of swallowing them.
- codex-lsp-adapter: import from vscode-languageserver/node.js (the bare
/node subpath does not resolve under ESM), add the missing
vscode-languageserver-textdocument dependency, drop unused deps,
replace any-typed handlers with LSP types, extract testable helpers
and cover them.
- learning-integration: fix invalid named imports from postgres (Sql),
move MySQL-style inline INDEX definitions out of CREATE TABLE into
CREATE INDEX (PostgreSQL syntax), drop ON CONFLICT on a non-unique
column, ?? instead of || so 0-values survive, drop unused deps.
Build & CI:
- Add the missing root tsconfig.json; wire adapters to the client via
project references (tsc --build) and give the client a prepare hook.
- Regenerate package-lock.json on Linux: the previous lockfile only
contained darwin-arm64 binaries for rollup/esbuild, so vitest and tsx
could not run on ubuntu CI at all.
- CI now builds client + adapters + learning-integration and runs their
test suites (hard-fail); gateway suite keeps its soft-pass because of
pre-existing failures in injection-defense/pii-redaction/scoring/
semantic-cache tests.
- Rewrite the integration test: gateway-dependent suites skip cleanly
when no gateway is reachable, the createEOPulseClient leftover is
gone, and TIP construction paths are covered without infrastructure.
Consistency:
- Replace the scrubbed 0000 port placeholder with a canonical default
(gateway 8787, ChatGPT adapter 8788) across server, client, compose,
Dockerfile (EXPOSE/healthcheck), .env.example, READMEs, and CORS
defaults; docker-compose port mapping "${PORT:-0}:${PORT:-0}" was
unusable.
- Update ADR-0005 with the implemented API and an implementation note.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01My8vF1RrR8hfrcD1evEXnW
… training, learned routing applied The learning stack existed but could not run autonomously: - Six tables the learning engine's jobs require (ban_candidates, few_shot_candidates, routing_candidates, ab_tests, prompt_candidates, learning_reports) were created by no migration — every scheduled job crashed on its first query. New migration 011 adds them (schemas derived from the jobs' actual SQL) plus column fixes: model_performance success_rate NUMERIC(6,4) for fraction semantics, confidence_avg NUMERIC(4,2) so a 10.00 average fits. - The boot-time migration runner skipped 010-feature-tables.sql, so the adaptive_routing table never existed either; 010 and 011 are now in the list, and init-db.sh applies all migrations instead of only 001. - getAdaptiveRecommendation was imported but never called: learned routing never influenced a single request. classifyAndRoute now applies the learned preferred model and fallback chain whenever the caller did not pin a model explicitly. Adaptive routing is on by default (ADAPTIVE_ROUTING_ENABLED=0 disables). - Adaptive recommendations are now persisted into adaptive_routing and warm-started at boot, so learned routing survives restarts; an empty learner pass keeps the previous map instead of wiping it. - The gateway-internal learning cycle wrote success_rate as a 0-100 percentage while its own detector and the insights route compare fractions; it now writes fractions, and the first cycle runs 2 minutes after boot instead of 6 hours later. - The learning service posted reload/report signals into the void: the gateway now implements POST /internal/reload-config (live routing-rules/models reload) and POST /internal/learning-report, guarded by the shared INTERNAL_SECRET header. - Few-shot curator negative examples never carried llm_call_id, so the dedup check missed them and every hourly run re-inserted the same rejections; the rejection's call_id is now stored. - New Dockerfile.learning + llm-learning service in docker-compose: the learning engine starts automatically alongside gateway and Postgres. README documents the loop and the container-mode caveat for YAML-writing jobs; .env.example gains INTERNAL_SECRET and the adaptive-routing knobs. Verified against a real PostgreSQL 16: all five migrations apply from a blank database, the learner trains on seeded traffic and persists the correct Pareto winner, warm-start restores it after restart, the internal endpoints enforce the secret, and ban-learner / few-shot-curator / routing-optimizer / learning-report all complete without errors for the first time. Six new unit tests cover the adaptive learner. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01My8vF1RrR8hfrcD1evEXnW
…e is now hard The router was built against a different config schema than the shipped YAML files: routing-rules.yaml uses a compact `rules:` format and models.yaml had no `tiers`/`fallback_chains` blocks, so `route()` and `routeByScore()` threw on every call — static routing, scored routing and the /v1/completion path could never work, and 15 tests encoding the intended behavior of the routing/security modules failed silently behind CI's soft pass. Router: - Normalize both config formats at load time: `rules:` or `routing_rules:` accepted, compact entries filled with defaults (callers, temperature, max_tokens per tier, output_format, validators), `reasoning` model tier mapped to large-tier budgets, built-in tier timeouts (fast 30s / medium 60s / large 180s) that models.yaml can override. - Dedicated fallback chains for the scorer tiers: code_generation walks gpt-4-turbo → deepseek-r1:32b → qwen2.5:32b → llama3.3:70b. - models.yaml gains the deepseek-r1:32b and gpt-4-turbo entries plus the fallback_chains block (moved from routing-rules.yaml defaults); routing-rules.yaml gains an explicit code_generation rule. - The learning-service routing optimizer now reads either top-level key and preserves the rest of the YAML document when it rewrites rules (previously a rewrite would have dropped every other section). Request scorer: - "Write a TypeScript function …" / "Create a React component …" style requests were invisible to the exact-phrase trie; a structural verb+artifact pattern now counts like a keyword match. - The code-generation override lifts score and confidence into the code_generation band so results are coherent for downstream consumers. Security modules: - injection-defense: "disregard the prior instructions" now matches (article variants), "pretend you don't have safety restrictions" matches noun forms, and the verbatim-repeat / everything-above system-prompt-leak probes are rated high so a single hit crosses the detection threshold. - pii-redaction: IBAN and credit-card rules run before the loose phone rules, which previously consumed digit groups inside them (an IBAN or card was redacted as a phone number fragment); the E.164 pattern now actually matches "+…" numbers (a word boundary can never precede "+"). Tests: - semantic-cache: the test's fake embedding vectors were scalar multiples of each other (cosine similarity 1.0 for any two seeds), inventing a false cache-hit; seeds now land in genuinely different directions. - Full gateway suite: 130 passed, 0 failed (10 live-integration tests skip without a running gateway). CI runs `vitest run` as a hard gate — the soft pass is gone. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01My8vF1RrR8hfrcD1evEXnW
…live, training sets build and export themselves Three learning outputs used to end in dead ends; all three now feed back into the running system: Learned banlist (ban-learner loop): - Promoted ban_candidates never reached validation: the banlist sync only pulls a remote CSV (placeholder URL), so everything the ban learner promoted was ignored. A new learned-banlist module loads promoted, non-rejected candidates from the database at boot and every 30 minutes, and banlist-checker includes them alongside the static EN/DE/auto lists (LEARNED_BANLIST_ENABLED=0 disables). Covered by six unit tests. Training corpus + export (fine-tuning loop): - learning_corpus had no writer, so the weekly fine-tuning trigger could never fire. The few-shot curator now records every high-confidence completion with a real input as a training example (deduped per call). - The trigger's INSERT used columns fine_tuning_runs does not have (task_type, epochs, lora_rank …) and would have crashed on first fire; it now matches the real schema, and instead of queueing into the void it exports the training set as JSONL to FINE_TUNING_EXPORT_DIR, marks the examples as consumed (included_in_run) and the run as 'exported' — ready for external LoRA tooling. Threshold configurable via FINE_TUNING_MIN_EXAMPLES. - checkFineTuningTrigger is exported for manual/ops invocation. Verified live against PostgreSQL 16: seeded 30 high-confidence reviewed completions → curator filled learning_corpus (30) → trigger created a run, exported 30 JSONL examples and marked them consumed; a seeded promoted ban candidate shows up in the gateway's learned banlist at boot. Full suite: gateway 136 passed, adapters + client 42 passed, zero failures. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01My8vF1RrR8hfrcD1evEXnW
…gacy bridges runnable
ESLint (the PR checklist demands `npx eslint packages/*/src`, but the repo
had no config, no dependency, and the CI lint job swallowed everything
with `|| true`):
- Add a flat config (eslint + typescript-eslint recommended) at the root,
fix all 42 findings, and make the CI lint job a hard gate.
- The lint pass surfaced a real functional bug: PII redaction never
restored the caller's response — restorePii was imported but never
called, so with REDACT_PII_MODE enabled callers received <EMAIL_001>
tokens instead of their data. The caller response is now restored,
while audit log, review queue, and response cache keep the redacted
form so PII never persists and cannot leak to other callers via cache
hits.
- Other findings fixed for real rather than silenced: dead imports
(unwired trackFallbackChain/reasoning-trace/post-hook imports in the
completion route, unused TLS/learning imports), error causes attached
to rethrown errors (preserve-caught-error), useless assignments
removed, empty catch blocks documented, a `Function`-typed pg-boss
call given a real signature, and regex escapes cleaned.
Orphaned root src/:
- src/pipeline/post-validator.ts was a diverged fragment of another
project (imports fastify types that don't exist, extensionless
imports) — removed.
- src/validation/nist-auth.ts + its 33 passing tests are real, working
code with no home: moved into packages/gateway/src/validation/ (with a
typed scrypt wrapper so the NIST cost parameters typecheck) and now
part of the gateway suite.
Legacy bridges:
- copilot-bridge and openai-bridge still carried the scrubbed 0000 port
placeholder: parseInt('0000') = port 0 meant random ports and dead
README instructions. Real defaults now: openai-bridge 8790,
copilot-bridge 8791 (wrapper) / 8792 (internal copilot-api), README
updated in all 18 places.
Full verification: builds + build-drift green, eslint clean, gateway 169
passed (now including nist-auth), adapters + client 42 passed.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01My8vF1RrR8hfrcD1evEXnW
renefichtmueller
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Five commits that take the gateway from "nothing TIP-related compiles" to a verified autonomous system:
createTIPClient/TIPClientConfigimplemented in@llm-gateway/client(both ADR-0005 config-object and legacy URL signatures); all three agent adapters (claude-code-bridge, chatgpt-api-adapter, codex-lsp-adapter) fixed to build, run, and pass hermetic tests; missing roottsconfig.jsonadded;package-lock.jsonregenerated on Linux (the old one only had darwin-arm64 binaries for rollup/esbuild — vitest/tsx could never run on ubuntu CI).ban_candidates,few_shot_candidates,routing_candidates,ab_tests,prompt_candidates,learning_reports); migration runner now includes 010+011; adaptive routing is consumed by the router (was imported but never called), persisted, warm-started after restarts, and on by default;POST /internal/reload-config+/internal/learning-reportendpoints exist now;llm-learningCompose service starts the learning engine automatically.route()/routeByScore()threw on every call; the loader now normalizes both formats. Request scorer detects "Write a TypeScript function …"-style prompts; injection-defense and PII-redaction detection gaps fixed (rule ordering ate IBANs/credit cards;\bbefore+can never match). All 15 previously failing tests pass; CI test soft-pass removed.learning_corpus; the fine-tuning trigger (whose INSERT didn't match the real schema) now exports ready-to-train JSONL sets and marks runsexported.restorePiiimported, never called) — fixed so callers get originals while audit/cache stay redacted. Orphaned rootsrc/resolved (broken fragment removed, workingnist-auth+ 33 tests moved into the gateway). Legacy copilot/openai bridges got real default ports (8790/8791/8792) instead of the scrubbed0000.Why
The TIP integration layer and the learning system existed on paper but could not run: consumers imported an API that wasn't exported, six DB tables had no migration, the router crashed on its own shipped configs, and CI's
|| truesoft-passes hid all of it. Goal: the gateway builds, routes, learns, and trains autonomously — verified, not assumed.How
0000placeholder: gateway 8787, ChatGPT adapter 8788, openai-bridge 8790, copilot-bridge 8791/8792 (all env-overridable).dist+ declarations) with project references; adapters build viatsc --build.X-Internal-Secret, ban-learner/few-shot-curator/routing-optimizer/learning-report jobs complete, fine-tuning trigger exports JSONL.routing-rules.yaml/prompt templates only reach the gateway's copy via shared volume or host mode — documented in README; their DB results flow regardless.Checklist
npm --workspace=packages/gateway run build)npx eslint packages/*/src)npm --workspace=packages/gateway test) — 169 gateway + 42 client/adapters, 0 failuresdist/)