Skip to content

README.md

Latest commit

 

History

History
106 lines (84 loc) · 4.18 KB

README.md

File metadata and controls

106 lines (84 loc) · 4.18 KB
Raw

Monitoring And Observability For LLM Services

This repository teaches how to instrument an LLM workflow with Langfuse, build useful views for service and model quality, and diagnose latency, cost, error, and answer quality regressions. Every workflow runs locally through auditable JSONL traces; Langfuse Cloud export is optional and uses the current observation-centric Python SDK.

Please use this repository as a template when starting your observability work.

Project Hub

Required Workflow

Work through the repository in this order:

  1. Generate local traces and inspect the root request, nested stages, model, token, cost, and quality fields.
  2. Export the same observations to Langfuse and build views for volume, latency, errors, tokens, cost, and quality.
  3. Compare releases and identify whether a regression comes from retrieval, generation, provider errors, or the service boundary.
  4. Add a deterministic quality check and interpret it alongside load evidence.
  5. Document masking, sampling, access, and retention decisions before live traffic.
flowchart LR
    A["LLM request"] --> B["Root observation"]
    B --> C["Retrieval span"]
    B --> D["Generation observation"]
    D --> E["Tokens and cost"]
    D --> F["Quality score"]
    B --> G["Dashboard and diagnosis"]
Loading

Deliverables

Your repository must include:

  • trace records with nested stages, model, token usage, cost, latency, release, environment, and quality evidence;
  • a Langfuse dashboard plan covering volume, p95 latency, errors, tokens, cost, and quality;
  • an incident diagnosis connected to individual traces and release metadata;
  • a documented masking and sampling policy; and
  • answers to the four checkpoints in the numbered implementation files.

Environment

The repository uses uv and the checked-in lockfile. The local workflow requires no Langfuse account or API key.

uv sync
cp .env.example .env
uv run python -m observability_lab summarize
uv run pytest

The commands use data/sample_traces.jsonl when no generated trace file exists. To send observations to Langfuse, configure LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, and LANGFUSE_BASE_URL, then run:

uv run python -m observability_lab run-demo --sink langfuse

Do not export raw prompts or outputs until their data classification and retention rules are explicit.

Learning Objectives

By completing this repository, you should be able to:

  • Instrument an LLM request with nested spans, generation usage, cost, and quality scores.
  • Build a dashboard for volume, latency, errors, tokens, cost, and quality.
  • Trace a retrieval-plus-generation workflow end to end and isolate the stage responsible for a regression.
  • Diagnose a production issue from trace evidence and interpret quality and load signals without conflating them.

Useful References