Merge pull request #57 from tma1-ai/greptime-batch-export-local

feat(batch-export): local-volume backend so object storage is fully optional (#48)

Author: killme2008

.env.dev.example

@@ -95,7 +95,12 @@ EMAIL_FROM_ADDRESS=""  # Defines the email address to use as the from address.
 SMTP_CONNECTION_URL=""  # Defines the connection url for smtp server.
 CLOUD_CRM_EMAIL=""  # Optional BCC address for usage threshold emails (e.g., for CRM integration like HubSpot)
 
-# S3 Batch Exports
+# Batch Exports
+## Backend: "local" writes the export to a shared volume and serves it via a signed web download
+## URL (no object storage needed); "s3" uploads to a bucket and hands out a presigned URL.
+# LANGFUSE_BATCH_EXPORT_STORAGE_BACKEND=local
+# LANGFUSE_BATCH_EXPORT_LOCAL_PATH=/tmp/langfuse/batch-exports
+# S3 Batch Exports (only when LANGFUSE_BATCH_EXPORT_STORAGE_BACKEND=s3)
 LANGFUSE_S3_BATCH_EXPORT_ENABLED=true
 LANGFUSE_S3_BATCH_EXPORT_BUCKET=langfuse
 LANGFUSE_S3_BATCH_EXPORT_ACCESS_KEY_ID=minio

.env.prod.example

@@ -161,7 +161,13 @@ OTEL_SERVICE_NAME="langfuse"
 # Defines the connection url for smtp server.
 # SMTP_CONNECTION_URL=
 
-# S3 Batch Exports
+# Batch Exports
+# Backend: "local" writes the export to a shared volume and serves it via a signed web download URL
+# (no object storage needed; the web download URL is built from NEXTAUTH_URL); "s3" uploads to a
+# bucket and hands out a presigned URL. The local path must be shared between web and worker.
+# LANGFUSE_BATCH_EXPORT_STORAGE_BACKEND=local
+# LANGFUSE_BATCH_EXPORT_LOCAL_PATH=/langfuse_batch_export_data
+# S3 Batch Exports (only when LANGFUSE_BATCH_EXPORT_STORAGE_BACKEND=s3)
 # LANGFUSE_S3_BATCH_EXPORT_ENABLED=
 # LANGFUSE_S3_BATCH_EXPORT_BUCKET=
 # LANGFUSE_S3_BATCH_EXPORT_ACCESS_KEY_ID=

Dockerfile.standalone

@@ -132,8 +132,8 @@ COPY --chown=openfuse:openfuse ./scripts/standalone-entrypoint.sh ./standalone-e
 RUN chmod +x ./standalone-entrypoint.sh
 
 # Default local blob-store mounts owned by the app user (see web/Dockerfile).
-RUN mkdir -p /langfuse_media_data /langfuse_event_data && \
-    chown openfuse:openfuse /langfuse_media_data /langfuse_event_data
+RUN mkdir -p /langfuse_media_data /langfuse_event_data /langfuse_batch_export_data && \
+    chown openfuse:openfuse /langfuse_media_data /langfuse_event_data /langfuse_batch_export_data
 
 USER openfuse
 

README.md

@@ -112,7 +112,7 @@ Full instructions for standalone, split web/worker images, and tag policy: [depl
 
 ## Architecture
 
-Postgres holds application and config data (users, projects, prompts, dataset definitions, API keys), unchanged from upstream Langfuse. GreptimeDB is the analytics event store: an append-only `raw_events` table as the source of truth, plus merged projection tables and indexed EAV side-tables that back metadata, tag, and tool filtering. Redis runs the BullMQ queues. Object storage (S3/MinIO) is optional for the default stack: media uploads, the OTel carrier, and the eval blob store default to local filesystem paths. Opt-in batch/blob exports still need an S3-compatible bucket.
+Postgres holds application and config data (users, projects, prompts, dataset definitions, API keys), unchanged from upstream Langfuse. GreptimeDB is the analytics event store: an append-only `raw_events` table as the source of truth, plus merged projection tables and indexed EAV side-tables that back metadata, tag, and tool filtering. Redis runs the BullMQ queues. Object storage (S3/MinIO) is optional for the default stack: media uploads, the OTel carrier, the eval blob store, and batch exports all default to local filesystem paths, so a stock deployment needs no bucket at all.
 
 Full write-up: [architecture](docs/architecture.md).
 

docker-compose.standalone.yml

@@ -38,6 +38,7 @@ services:
     volumes:
       - langfuse_media_data:/langfuse_media_data
       - langfuse_event_data:/langfuse_event_data
+      - langfuse_batch_export_data:/langfuse_batch_export_data
     environment:
       # --- core ---
       NEXTAUTH_URL: ${NEXTAUTH_URL:-http://localhost:3000}
@@ -64,6 +65,8 @@ services:
       LANGFUSE_MEDIA_LOCAL_PATH: ${LANGFUSE_MEDIA_LOCAL_PATH:-/langfuse_media_data}
       LANGFUSE_EVENT_STORAGE_BACKEND: ${LANGFUSE_EVENT_STORAGE_BACKEND:-local}
       LANGFUSE_EVENT_LOCAL_PATH: ${LANGFUSE_EVENT_LOCAL_PATH:-/langfuse_event_data}
+      LANGFUSE_BATCH_EXPORT_STORAGE_BACKEND: ${LANGFUSE_BATCH_EXPORT_STORAGE_BACKEND:-local}
+      LANGFUSE_BATCH_EXPORT_LOCAL_PATH: ${LANGFUSE_BATCH_EXPORT_LOCAL_PATH:-/langfuse_batch_export_data}
       # --- optional initial bootstrap org/project/user (web only) ---
       LANGFUSE_INIT_ORG_ID: ${LANGFUSE_INIT_ORG_ID:-}
       LANGFUSE_INIT_ORG_NAME: ${LANGFUSE_INIT_ORG_NAME:-}
@@ -144,5 +147,7 @@ volumes:
     driver: local
   langfuse_event_data:
     driver: local
+  langfuse_batch_export_data:
+    driver: local
   langfuse_redis_data:
     driver: local

docker-compose.yml

@@ -28,8 +28,12 @@ services:
     volumes: &langfuse-blob-volumes
       - langfuse_media_data:/langfuse_media_data
       - langfuse_event_data:/langfuse_event_data
+      - langfuse_batch_export_data:/langfuse_batch_export_data
     environment: &langfuse-worker-env
       NEXTAUTH_URL: ${NEXTAUTH_URL:-http://localhost:3000}
+      # Shared with web so the worker can mint signed local batch-export download
+      # tokens that the web download route verifies with the same secret.
+      NEXTAUTH_SECRET: ${NEXTAUTH_SECRET:-mysecret} # CHANGEME
       DATABASE_URL: ${DATABASE_URL:-postgresql://postgres:postgres@postgres:5432/postgres} # CHANGEME
       SALT: ${SALT:-mysalt} # CHANGEME
       ENCRYPTION_KEY: ${ENCRYPTION_KEY:-0000000000000000000000000000000000000000000000000000000000000000} # CHANGEME: generate via `openssl rand -hex 32`
@@ -68,6 +72,11 @@ services:
       LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT: ${LANGFUSE_S3_MEDIA_UPLOAD_ENDPOINT:-http://localhost:9090}
       LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE: ${LANGFUSE_S3_MEDIA_UPLOAD_FORCE_PATH_STYLE:-true}
       LANGFUSE_S3_MEDIA_UPLOAD_PREFIX: ${LANGFUSE_S3_MEDIA_UPLOAD_PREFIX:-media/}
+      # Batch exports default to the local-file backend (shared volume + signed
+      # web download URL), so no bucket is needed. Set to "s3" + start the minio
+      # profile (or your own bucket) to use object storage instead.
+      LANGFUSE_BATCH_EXPORT_STORAGE_BACKEND: ${LANGFUSE_BATCH_EXPORT_STORAGE_BACKEND:-local}
+      LANGFUSE_BATCH_EXPORT_LOCAL_PATH: ${LANGFUSE_BATCH_EXPORT_LOCAL_PATH:-/langfuse_batch_export_data}
       LANGFUSE_S3_BATCH_EXPORT_ENABLED: ${LANGFUSE_S3_BATCH_EXPORT_ENABLED:-false}
       LANGFUSE_S3_BATCH_EXPORT_BUCKET: ${LANGFUSE_S3_BATCH_EXPORT_BUCKET:-langfuse}
       LANGFUSE_S3_BATCH_EXPORT_PREFIX: ${LANGFUSE_S3_BATCH_EXPORT_PREFIX:-exports/}
@@ -104,7 +113,6 @@ services:
     volumes: *langfuse-blob-volumes
     environment:
       <<: *langfuse-worker-env
-      NEXTAUTH_SECRET: ${NEXTAUTH_SECRET:-mysecret} # CHANGEME
       LANGFUSE_INIT_ORG_ID: ${LANGFUSE_INIT_ORG_ID:-}
       LANGFUSE_INIT_ORG_NAME: ${LANGFUSE_INIT_ORG_NAME:-}
       LANGFUSE_INIT_PROJECT_ID: ${LANGFUSE_INIT_PROJECT_ID:-}
@@ -212,6 +220,8 @@ volumes:
     driver: local
   langfuse_event_data:
     driver: local
+  langfuse_batch_export_data:
+    driver: local
   langfuse_minio_data:
     driver: local
   langfuse_redis_data:

docs/architecture.md

@@ -49,7 +49,7 @@ BullMQ job queues for ingestion, evals, exports, and reconciliation. Unchanged f
 
 ### Object storage (optional)
 
-With the event store in GreptimeDB, S3 or MinIO is no longer required to ingest. Media uploads, the OTel ingestion carrier, and the eval blob store support a local-filesystem backend, and the bundled Compose files default them to it. Opt-in batch/blob _exports_ still need an S3-compatible bucket.
+With the event store in GreptimeDB, S3 or MinIO is no longer required. Media uploads, the OTel ingestion carrier, the eval blob store, and batch exports all support a local-filesystem backend, and the bundled Compose files default them to it. Batch exports on the local backend stream from a shared volume via a signed, time-limited download URL instead of an S3 presigned URL, so a stock deployment needs no object storage at all.
 
 ## Write path: source of truth and replay
 

docs/deployment.md

@@ -51,31 +51,33 @@ The fork ships a small server config at `docker/greptimedb/config.toml`, mounted
 
 ### Object storage is optional
 
-Ingestion and eval-generated scores persist to GreptimeDB `raw_events`, not to a blob store. The remaining object-storage consumers support a local-file backend, so a stock deployment needs **no** MinIO/S3:
+Ingestion and eval-generated scores persist to GreptimeDB `raw_events`, not to a blob store. Every remaining object-storage consumer supports a local-file backend, so a stock deployment needs **no** MinIO/S3:
 
-| Variable                         | App default | Bundled Compose | Local backend                         |
-| -------------------------------- | ----------- | --------------- | ------------------------------------- |
-| `LANGFUSE_MEDIA_STORAGE_BACKEND` | `s3`        | `local`         | `local` + `LANGFUSE_MEDIA_LOCAL_PATH` |
-| `LANGFUSE_EVENT_STORAGE_BACKEND` | `s3`        | `local`         | `local` + `LANGFUSE_EVENT_LOCAL_PATH` |
+| Variable                                | App default | Bundled Compose | Local backend                                |
+| --------------------------------------- | ----------- | --------------- | -------------------------------------------- |
+| `LANGFUSE_MEDIA_STORAGE_BACKEND`        | `s3`        | `local`         | `local` + `LANGFUSE_MEDIA_LOCAL_PATH`        |
+| `LANGFUSE_EVENT_STORAGE_BACKEND`        | `s3`        | `local`         | `local` + `LANGFUSE_EVENT_LOCAL_PATH`        |
+| `LANGFUSE_BATCH_EXPORT_STORAGE_BACKEND` | `s3`        | `local`         | `local` + `LANGFUSE_BATCH_EXPORT_LOCAL_PATH` |
 
-The application default for these variables is `s3`, but this repo's `docker-compose.yml` overrides both to `local` (`${...:-local}`), so the bundled stack starts with no object store. `LANGFUSE_EVENT_STORAGE_BACKEND` covers both the OTel carrier and the eval blob store; with `local` they share a filesystem volume, so web and worker must mount the same `LANGFUSE_EVENT_LOCAL_PATH` (the Compose files wire a shared `langfuse_event_data` volume). Only opt-in batch/blob **exports** still require an S3-compatible bucket. The Compose files default both backends to `local` and put MinIO behind a `s3` profile (`docker compose --profile s3 up`), so the default stack starts no object store.
+The application default for these variables is `s3`, but this repo's `docker-compose.yml` overrides them to `local` (`${...:-local}`), so the bundled stack starts with no object store. Each `local` backend shares a filesystem volume between web and worker (the Compose files wire `langfuse_event_data` and `langfuse_batch_export_data`); `LANGFUSE_EVENT_STORAGE_BACKEND` covers both the OTel carrier and the eval blob store. Batch exports on the `local` backend are served by an authenticated web download route (signed, time-limited token, re-validated against the export row) instead of an S3 presigned URL, so the link in the export email and the exports page works without a bucket. The Compose files put MinIO behind a `s3` profile (`docker compose --profile s3 up`), so the default stack starts no object store.
 
 ### Data directories and persistent volumes
 
 The Compose files use Docker named volumes for every stateful path. Treat these as the deployment's data directory set:
 
-| Volume                       | Mounted path                  | Stores                                                                 |
-| ---------------------------- | ----------------------------- | ---------------------------------------------------------------------- |
-| `langfuse_postgres_data`     | `/var/lib/postgresql/data`    | Postgres application state: users, projects, API keys, prompts, config |
-| `langfuse_greptimedb_data`   | `/greptimedb_data`            | GreptimeDB analytics store: traces, observations, scores, dashboards   |
-| `langfuse_redis_data`        | `/data`                       | Redis queue state                                                       |
-| `langfuse_media_data`        | `/langfuse_media_data`        | Local media uploads when `LANGFUSE_MEDIA_STORAGE_BACKEND=local`        |
+| Volume                       | Mounted path                  | Stores                                                                        |
+| ---------------------------- | ----------------------------- | ----------------------------------------------------------------------------- |
+| `langfuse_postgres_data`     | `/var/lib/postgresql/data`    | Postgres application state: users, projects, API keys, prompts, config        |
+| `langfuse_greptimedb_data`   | `/greptimedb_data`            | GreptimeDB analytics store: traces, observations, scores, dashboards          |
+| `langfuse_redis_data`        | `/data`                       | Redis queue state                                                             |
+| `langfuse_media_data`        | `/langfuse_media_data`        | Local media uploads when `LANGFUSE_MEDIA_STORAGE_BACKEND=local`               |
 | `langfuse_event_data`        | `/langfuse_event_data`        | Local OTel carrier and eval blobs when `LANGFUSE_EVENT_STORAGE_BACKEND=local` |
-| `langfuse_minio_data`        | `/data` in `minio`            | Optional MinIO bucket data when the `s3` profile is enabled             |
+| `langfuse_batch_export_data` | `/langfuse_batch_export_data` | Local batch-export files when `LANGFUSE_BATCH_EXPORT_STORAGE_BACKEND=local`   |
+| `langfuse_minio_data`        | `/data` in `minio`            | Optional MinIO bucket data when the `s3` profile is enabled                   |
 
 Runtime logs are not written to a separate application log directory by default. Web, worker, the standalone supervisor, Postgres, Redis, and GreptimeDB all write to container stdout/stderr; collect them through `docker compose logs` or your Docker logging driver. If you enable GreptimeDB file logging or replace Docker named volumes with bind mounts, keep the log directory outside the container's writable layer and include it in the same backup/retention plan as `langfuse_greptimedb_data`.
 
-Do not run `docker compose down -v` on a real deployment unless you intentionally want to delete the service data. `docker compose down` removes containers and networks but keeps the named volumes; `down -v` removes them. If you replace the named volumes with bind mounts, keep the same container paths above and make sure the app containers can write to the media/event paths. In the split topology, both `langfuse-web` and `langfuse-worker` must mount the same `langfuse_media_data` and `langfuse_event_data` storage.
+Do not run `docker compose down -v` on a real deployment unless you intentionally want to delete the service data. `docker compose down` removes containers and networks but keeps the named volumes; `down -v` removes them. If you replace the named volumes with bind mounts, keep the same container paths above and make sure the app containers can write to the media/event paths. In the split topology, both `langfuse-web` and `langfuse-worker` must mount the same `langfuse_media_data`, `langfuse_event_data`, and `langfuse_batch_export_data` storage.
 
 ## 2. Migrations run automatically on startup
 

docs/greptimedb-migration/07-deployment.md

@@ -40,14 +40,14 @@ section.
 
 Ingestion and eval-generated scores persist to GreptimeDB `raw_events`, not to a
 blob store. The remaining object-storage consumers — media uploads, the OTel
-ingestion carrier, and the eval observation blob store — all support a local-file
-backend, so a stock deployment needs **no** MinIO/S3 (only opt-in batch/blob
-exports still require an S3-compatible bucket):
-
-| Variable                         | Default | Set to `local` for no object storage  |
-| -------------------------------- | ------- | ------------------------------------- |
-| `LANGFUSE_MEDIA_STORAGE_BACKEND` | `s3`    | `local` + `LANGFUSE_MEDIA_LOCAL_PATH` |
-| `LANGFUSE_EVENT_STORAGE_BACKEND` | `s3`    | `local` + `LANGFUSE_EVENT_LOCAL_PATH` |
+ingestion carrier, the eval observation blob store, and batch exports — all
+support a local-file backend, so a stock deployment needs **no** MinIO/S3:
+
+| Variable                                | Default | Set to `local` for no object storage         |
+| --------------------------------------- | ------- | -------------------------------------------- |
+| `LANGFUSE_MEDIA_STORAGE_BACKEND`        | `s3`    | `local` + `LANGFUSE_MEDIA_LOCAL_PATH`        |
+| `LANGFUSE_EVENT_STORAGE_BACKEND`        | `s3`    | `local` + `LANGFUSE_EVENT_LOCAL_PATH`        |
+| `LANGFUSE_BATCH_EXPORT_STORAGE_BACKEND` | `s3`    | `local` + `LANGFUSE_BATCH_EXPORT_LOCAL_PATH` |
 
 `LANGFUSE_EVENT_STORAGE_BACKEND` covers both the OTel carrier and the eval blob
 store; with `local` they share a filesystem volume, so the API and worker must

docs/known-limitations.md

@@ -18,7 +18,7 @@ The web and standalone entrypoints apply the GreptimeDB schema on startup (gated
 
 ### Object storage is optional, not gone
 
-Ingestion needs no object store — traces, observations, and scores persist to GreptimeDB `raw_events`. Media uploads, the OTel carrier, and the eval blob store default to local filesystem volumes in the bundled Compose files. But opt-in batch/blob **exports** still require an S3-compatible bucket. See [deployment](deployment.md).
+Ingestion needs no object store — traces, observations, and scores persist to GreptimeDB `raw_events`. Media uploads, the OTel carrier, the eval blob store, and batch exports all default to local filesystem volumes in the bundled Compose files, so a stock deployment runs with no S3/MinIO at all. Object storage stays fully supported (set the `*_STORAGE_BACKEND` variables to `s3`) for multi-node deployments, where a shared object store is the simpler way to share blobs across replicas. See [deployment](deployment.md).
 
 ## Source-of-truth exception