Skip to content

operations.md

Latest commit

 

History

History
80 lines (49 loc) · 10.2 KB

operations.md

File metadata and controls

80 lines (49 loc) · 10.2 KB
Raw

Operations

Openfuse runs the Langfuse app against two stores: Postgres for app/config data and GreptimeDB for the analytics event store. Operating it is mostly operating those two databases. This page covers the fork-specific operational notes and points to the upstream runbooks for everything generic — it does not duplicate them.

For first-time setup (env, automatic migrations, Compose, images) see deployment. For local dev see development.

Configuration

Configuration is split by store, so there is no single config file to learn:

  • Fork-specific GREPTIME_*, retention (LANGFUSE_GREPTIME_TTL), object-storage, and the migration toggles are documented in deployment · Configuration; the source of truth is packages/shared/src/env.ts.
  • Everything else (auth/SSO, Postgres, Redis, secrets, headers, scaling) is upstream Langfuse and unchanged — see Langfuse · Configuration.
  • GreptimeDB server config ships at docker/greptimedb/config.toml (mounted read-only, passed via --config-file): GreptimeDB defaults plus commented tuning hints. Everything else (data dir, object storage, WAL, table options) is GreptimeDB's own — see GreptimeDB · Configuration.
  • GreptimeDB authentication is off by default and turns on when GREPTIME_PASSWORD is set: the greptimedb container then enforces a static user and the app authenticates with the matching GREPTIME_USER/GREPTIME_PASSWORD. Set both for any real deployment — see deployment · GreptimeDB authentication.

Monitoring

The worker samples per-table GreptimeDB region statistics every 60 s (GreptimeStatsRunner, gated by LANGFUSE_GREPTIME_STATS_ENABLED, period LANGFUSE_GREPTIME_STATS_INTERVAL_MS) and emits these gauges, all tagged by table:

Metric Meaning
langfuse.greptime.sst_files_max Per-region maximum SST count; hits the 384 wall first — the one to alert on.
langfuse.greptime.sst_files Sum of SST files across the table's regions.
langfuse.greptime.region_rows Row count.
langfuse.greptime.disk_size On-disk bytes.
langfuse.greptime.memtable_size In-memory (un-flushed) bytes.

For the database itself, use GreptimeDB's own tooling: check DB status, self-monitoring/metrics, and slow queries. Application-level health, logs, and tracing for the Langfuse web/worker are unchanged from Langfuse · Self-hosting.

Performance: compaction

The one performance lever for the GreptimeDB read path is SST compaction, not indexing or query shape. By-type dashboard queries scan a time range and group by key; latency is dominated by how many SST files the scan has to merge. Measured on the same query (GreptimeDB 1.1.x, ~3.5M observations): 1022 un-compacted SST files → 9.6 s; after compact_table (1 file) → 0.2 s. A key skipping index does not help, so do not add one.

GreptimeDB enforces a hard ceiling of 384 SST files per region: above it even count(*) fails with Too many files (max allowed: 384) until background compaction catches up. The writer flushes roughly every second under load, so high ingest or a bulk backfill produces small SSTs fast.

After a large backfill, compact the hot tables once. Fleet reconciliation replays history through the write path and can land thousands of small SSTs (measured: ~2.5M observations → ~4032 SST files on observations_usage_cost), enough to trip the wall. After the backfill drains, run over the MySQL wire (:4002) against GREPTIME_DB:

ADMIN compact_table('observations_usage_cost', 'strict_window', 86400);
ADMIN compact_table('observations',            'strict_window', 86400);

strict_window with an explicit window (86400 s = 1 day) compacts within day-aligned windows; prefer a real window over a bare 0, which collapses the whole table in one pass and is disruptive on a long-TTL production table. Run fire-and-forget and watch sst_files_max drop back to single digits. The hot tables are the ingest-heavy ones — observations, observations_usage_cost, and the observation EAV side-tables — then traces / scores and their EAV tables.

Alert on langfuse.greptime.sst_files_max approaching 384 (e.g. warn at ~200). A steady climb with no backfill in flight means ingest is outrunning background compaction; tune the table-level TWCS options (compaction.twcs.*) rather than relying on repeated manual compaction.

Background: GreptimeDB compaction and performance-tuning tips. The deep fork-specific runbook with the scale evidence is greptimedb-migration/08-compaction-runbook.md.

Performance: ingestion drain

The worker rebuilds each entity's projection from its full raw_events history on every event, so drain throughput is bound by that read plus the projection/EAV write. A few fork-specific knobs tune it; defaults are sized for a single node (source of truth: worker/src/env.ts):

  • Encode off-load — LANGFUSE_GREPTIME_FLUSH_WORKER_POOL_SIZE (default 4) sizes the worker_threads pool that runs the GreptimeDB ingester's protobuf encode + gRPC write off the worker event loop (the synchronous encode otherwise starves every job's raw_events read). Keep it >= LANGFUSE_GREPTIME_MAX_CONCURRENT_FLUSHES (default 4); each pool worker holds its own gRPC client.
  • raw_events flush — LANGFUSE_GREPTIME_RAW_EVENTS_FLUSH_ENABLED (default true) and LANGFUSE_GREPTIME_RAW_EVENTS_FLUSH_INTERVAL_MS (default 90000) run ADMIN flush_table('raw_events') on a timer. The per-event point read is an O(memtable) scan while data sits unflushed; flushing keeps it on prunable SSTs (measured ~20× faster). Lower the interval for read-heavy drains at the cost of more, smaller SSTs (more compaction). It is scoped to raw_events so other tables keep GreptimeDB's engine-global auto_flush_interval.
  • Rebuild coalescing — LANGFUSE_INGESTION_COALESCE_REBUILDS (default true) and LANGFUSE_INGESTION_COALESCE_WATERMARK_TTL_SECONDS (default 300) skip a queued job's read + rebuild when a prior idempotent rebuild already covered all of its events — the dominant redundancy when draining a backlog. Requires Redis.

For measured drain / query-latency / storage numbers against upstream Langfuse on ClickHouse, see greptimedb-migration/10-benchmark-report.md.

Capacity planning and retention

Retention is database-level TTL: LANGFUSE_GREPTIME_TTL (default 730d) is applied at startup via ALTER DATABASE ... SET 'ttl', covering every table at once. To change it, set the env and restart.

The bundled single-node Compose stack stores GreptimeDB data on local disk (langfuse_greptimedb_data). GreptimeDB can also use object storage with local disk as cache, which lets storage scale independently of compute when you move beyond the default standalone setup. For sizing disk, object storage, and compute, see GreptimeDB · Capacity plan. For scaling the Langfuse web/worker tier, see Langfuse · Scaling.

Backup and disaster recovery

Two stores to protect:

GreptimeDB's raw_events is the analytics source of truth, so a restored raw_events can rebuild every projection by replay. Object storage already gives the analytics data cloud-level durability when GreptimeDB is backed by S3.

Maintenance and upgrades

  • GreptimeDB: maintenance mode and version upgrades.
  • Openfuse app: roll the tma1ai/openfuse-* images forward by pinning a newer tag and re-running docker compose up -d --pull always (see deployment). The web/standalone entrypoint re-applies the Postgres and GreptimeDB migrations on startup, so an upgrade that ships new schema migrates automatically; the GreptimeDB runner is idempotent and re-applies the full set on every start. Langfuse-specific upgrade notes: Langfuse · Upgrade.