Ollive Inferlog

Lightweight inference logging for LLM apps: a streaming chatbot, an auto-instrumenting Python SDK, an event-based ingestion pipeline, and a dashboard for metrics and traces.

Built for the Ollive.ai take-home assignment.

Current scope

What this repo ships today:

Area	What it does
Chatbot	Multi-turn chat with a 12-message context window, SSE streaming, cancel, resume, and provider/model selection
SDK (inferlog)	InferenceClient wraps OpenAI, Anthropic, Gemini, and DeepSeek; captures latency, tokens, status, and redacted previews
Auto-instrumentation	inferlog.instrument() patches OpenAI/Anthropic async clients so existing calls are logged without code changes
Ingestion	Validates payloads, returns 202, enqueues on Redis; a worker persists to Postgres
Traces	Per-request span waterfall (chatbot → SDK → provider → ingest) in the UI
Metrics	Latency percentiles (p50/p95/p99), throughput, and error rates by provider
Frontend	Conversation list, resume, in-flight cancel, metrics overview, and traces
PII	SDK redacts email, phone, SSN, card numbers, and API-key prefixes before shipping
Deploy	Docker Compose for local run; Kubernetes manifests under k8s/

What is out of scope for this MVP: auth, multi-tenant isolation, full payload retention, and hosted OpenTelemetry export.

System design

┌─────────────┐     inferlog SDK      ┌──────────────┐    Redis queue    ┌─────────┐
│  React UI   │ ──► Chatbot API ─────►│  LogShipper  │ ────────────────► │ Worker  │
└─────────────┘     (streaming SSE)   └──────┬───────┘                   └────┬────┘
                                             │ POST /v1/logs                     │
                                             ▼                                   ▼
                                      ┌──────────────┐                   ┌──────────────┐
                                      │  Ingestion   │                   │  PostgreSQL  │
                                      │  API         │                   │  logs + chat │
                                      └──────────────┘                   └──────────────┘

Logging never blocks the user-visible token stream. The SDK ships logs asynchronously; the chat path only talks to the LLM provider.

Components

Service	Role
frontend	Vite + React UI on port 3000 (nginx in Compose)
chatbot	FastAPI on 8000 — conversations, SSE chat, cancel
ingestion	FastAPI on 8001 — logs, metrics, traces
worker	Same image as ingestion; BRPOP Redis → insert Postgres
postgres	conversations, messages, inference_logs
redis	List queue inferlog:events

Request path (one chat turn)

1. UI POSTs to chatbot; chatbot starts a trace_id and records spans for load-history / context-build.
2. InferenceClient.complete() or .stream() calls the provider, times the call, and attaches SDK spans.
3. SDK PII-redacts input/output previews (500 chars by default) and fire-and-forgets POST /v1/logs.
4. Ingestion validates with Pydantic, skips duplicate log_id, LPUSHes to Redis, returns 202.
5. Worker BRPOPs and writes inference_logs (JSONB metadata holds trace_id + spans).
6. Metrics and Traces APIs read from that table; the UI polls them.

Sync fallback: POST /v1/logs/sync writes Postgres directly (tests / debugging).

Schema

Table	Purpose
conversations	Session — provider, model, status
messages	Full user/assistant turns (source of truth for chat)
inference_logs	One row per LLM call — latency, tokens, previews, status, JSONB metadata

Tradeoffs

• Previews in logs, full text in messages — keeps the log table small.
• Async ingest (202 + queue) — chat latency stays independent of DB writes.
• Denormalized provider/model on conversations — list view without joins.
• JSONB metadata — traces/spans without a dedicated spans table (MVP).

Instrumentation modes

1. Wrapper — InferenceClient used by the chatbot. Full control over streaming, cancel, and spans.
2. Auto-instrument — monkey-patches OpenAI/Anthropic async create(). Zero changes for apps already on those SDKs.

Failure handling

Failure	Behavior
Ingestion down	SDK keeps up to 256 logs in memory; drops oldest
Redis down	Ingestion 500; SDK treats it as a ship failure and buffers
Worker crash	Events stay in Redis (at-least-once)
Duplicate log_id	Skip enqueue / skip insert
Provider error	Log status=error, re-raise to the caller
User cancel	Abort stream, log status=cancelled with partial preview

Deeper notes: ARCHITECTURE.md.

Future scope

• Idempotent ingest with explicit dedup keys and a dead-letter queue
• OpenTelemetry export so traces can land in Jaeger / Grafana Tempo
• Batch log shipping with backoff/jitter and a persistent offline buffer
• Ingestion API keys, auth on the UI, and per-tenant isolation
• Time-partitioned inference_logs plus retention/TTL
• Move in-memory cancel flags to Redis so chatbot replicas stay consistent
• Alembic migrations instead of create_all on startup
• Hosted demo with seeded metrics

Quick start

cp .env.example .env
# Add at least one provider key, e.g. OPENAI_API_KEY=sk-...
docker compose up --build

Open http://localhost:3000

Service	URL
Frontend	http://localhost:3000
Chatbot	http://localhost:8000/docs
Ingestion	http://localhost:8001/docs

.env is gitignored. Copy from .env.example and keep keys local.

Demo

1. Start a chat and pick a provider.
2. Send messages (streaming). Use Cancel mid-generation.
3. Resume a past conversation from the sidebar.
4. Open Metrics for latency / throughput / errors, then Traces for the waterfall.

SDK usage

Wrapper client

from inferlog import InferenceClient

client = InferenceClient(
    provider="openai",
    conversation_id="conv-123",
    model="gpt-4o-mini",
)

result = await client.complete([
    {"role": "user", "content": "Hello"},
])
print(result["content"])

Streaming with cancel:

async for chunk in client.stream(messages, on_cancel=lambda: cancelled):
    print(chunk, end="")

Auto-instrumentation

from inferlog import instrument

instrument(ingestion_url="http://localhost:8001/v1/logs")
# Existing OpenAI/Anthropic async calls are now logged

Local development (without full Compose)

# Postgres + Redis: docker compose up postgres redis
cd services/ingestion && pip install -r requirements.txt && uvicorn app.main:app --port 8001
cd services/ingestion && python -m app.worker
cd services/chatbot && pip install ./sdk/inferlog -r requirements.txt && uvicorn app.main:app --port 8000
cd frontend && npm install && npm run dev

License

MIT