Reliable LLM Backend Gateway — proof of approach

Small, dependency-light proof for the LaborX AI/LLM Engineer — Backend & API Integration job.

It demonstrates the production concerns that sit between a mobile product and an external model provider:

• provider isolation behind a small protocol;
• structured JSON validation before application code sees a response;
• bounded conversational memory;
• an allow-listed tool registry with validated arguments;
• per-request token budgets and atomic usage accounting;
• retries only for transient provider failures;
• request correlation IDs without logging prompts, secrets, or personal data;
• deterministic tests that do not call a paid API.

The proof deliberately uses no real provider key. DemoProvider is a deterministic stand-in, so the design can be reviewed and tested safely. A production adapter for OpenAI, Anthropic, Gemini, or another provider can be added without changing the application-facing contract.

Run the proof

python3 demo.py
python3 -m unittest -v

Optional HTTP layer

api.py exposes the same gateway through FastAPI if FastAPI and Uvicorn are installed:

python3 -m pip install fastapi uvicorn
uvicorn api:app --reload

Then send a request:

curl -s http://127.0.0.1:8000/v1/triage \
  -H 'content-type: application/json' \
  -d '{"conversation_id":"demo-1","message":"Payment failed twice","max_tokens":300}'

Production next step

For the real MVP I would first agree a written contract for:

1. the exact JSON schema the mobile client consumes;
2. allowed tools and their authorization boundaries;
3. data retention and redaction rules;
4. latency, availability, and cost ceilings;
5. fallback behaviour when the primary provider is unavailable.

That keeps the first implementation lean while leaving provider choice, storage, and observability replaceable.