This repository teaches how to instrument an LLM workflow with Langfuse, build useful views for service and model quality, and diagnose latency, cost, error, and answer quality regressions. Every workflow runs locally through auditable JSONL traces; Langfuse Cloud export is optional and uses the current observation-centric Python SDK.
Please use this repository as a template when starting your observability work.
- 01-trace-the-request.md: Understand the trace schema and sensitive-data boundary.
- 02-langfuse-dashboard.md: Export observations and design actionable dashboard widgets.
- 03-diagnose-an-incident.md: Compare releases and locate a quality and latency regression.
- 04-quality-and-load.md: Combine sampled quality signals with service load evidence.
- assets/README.md: visual assets and source credits.
- data/sample_traces.jsonl: Deterministic incident dataset for the local workflow.
- src/observability_lab/: Trace, analysis, and CLI implementation.
Work through the repository in this order:
- Generate local traces and inspect the root request, nested stages, model, token, cost, and quality fields.
- Export the same observations to Langfuse and build views for volume, latency, errors, tokens, cost, and quality.
- Compare releases and identify whether a regression comes from retrieval, generation, provider errors, or the service boundary.
- Add a deterministic quality check and interpret it alongside load evidence.
- Document masking, sampling, access, and retention decisions before live traffic.
flowchart LR
A["LLM request"] --> B["Root observation"]
B --> C["Retrieval span"]
B --> D["Generation observation"]
D --> E["Tokens and cost"]
D --> F["Quality score"]
B --> G["Dashboard and diagnosis"]
Your repository must include:
- trace records with nested stages, model, token usage, cost, latency, release, environment, and quality evidence;
- a Langfuse dashboard plan covering volume, p95 latency, errors, tokens, cost, and quality;
- an incident diagnosis connected to individual traces and release metadata;
- a documented masking and sampling policy; and
- answers to the four checkpoints in the numbered implementation files.
The repository uses uv and the checked-in lockfile. The local workflow requires no Langfuse account or API key.
uv sync
cp .env.example .env
uv run python -m observability_lab summarize
uv run pytestThe commands use data/sample_traces.jsonl when no
generated trace file exists. To send observations to Langfuse, configure
LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, and LANGFUSE_BASE_URL, then run:
uv run python -m observability_lab run-demo --sink langfuseDo not export raw prompts or outputs until their data classification and retention rules are explicit.
By completing this repository, you should be able to:
- Instrument an LLM request with nested spans, generation usage, cost, and quality scores.
- Build a dashboard for volume, latency, errors, tokens, cost, and quality.
- Trace a retrieval-plus-generation workflow end to end and isolate the stage responsible for a regression.
- Diagnose a production issue from trace evidence and interpret quality and load signals without conflating them.