feat(batch-export): local-volume backend so object storage is fully optional (#48) - #57
Merged
Merged
Conversation
…ptional Batch exports were the last hard S3 dependency (issue #48). Add a LANGFUSE_BATCH_EXPORT_STORAGE_BACKEND=s3|local selector mirroring the event / media backends. On the local backend the worker writes the export to a shared filesystem volume and mints a signed, time-limited download token; an authenticated web route (/api/public/batch-exports/[id]/download) verifies the token and re-validates the batchExport row (project scope, COMPLETED, expiry) before streaming the file, replacing the S3 presigned URL. The signing helper lives in shared so the worker can mint and the web route can verify with the same secret (NEXTAUTH_SECRET ?? SALT); NEXTAUTH_SECRET is added to shared env and promoted to the shared compose env so web and worker resolve the same value. Compose files default the backend to local with a shared langfuse_batch_export_data volume, so a stock deployment needs no object store. Removes the 'batch exports still need an S3 bucket' caveat from the docs.
killme2008
force-pushed
the
greptime-batch-export-local
branch
from
June 24, 2026 09:08
5ed5d5b to
5315b1c
Compare
Local batch exports mint HMAC-signed download tokens and URLs in the worker. Web verifies tokens with NEXTAUTH_SECRET (required in production) and reaches links via NEXTAUTH_URL. The worker env schema validated neither, so a split deployment could boot a worker that signs tokens with a different secret (every download 400s) or mints unreachable http://localhost:3000 links, both silently. Require NEXTAUTH_SECRET whenever the local backend is selected, and NEXTAUTH_URL in production, so misconfiguration fails at boot instead of at download time.
The web download route streams the export file from LANGFUSE_BATCH_EXPORT_LOCAL_PATH. The web env left that path optional, so a 'local' backend with no path booted fine but 500'd every download while the worker still completed exports and stored signed URLs — easy to hit in split deployments where worker and web envs diverge. Validate it at boot like the worker does. t3-env exposes no cross-field hook, so the refine reads the sibling backend from process.env directly.
…lume Address PR #57 review: - Encode Content-Disposition with an ASCII-only filename fallback plus a UTF-8 filename*, mirroring the trace download route, so a crafted export file name cannot break or inject the header. - Document the langfuse_batch_export_data volume in docs/deployment.md (volumes table + shared-mount note).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #48.
Batch exports were the last hard S3 dependency. This adds a local-file backend so a stock deployment needs no object storage at all.
What changed
LANGFUSE_BATCH_EXPORT_STORAGE_BACKEND=s3|local+LANGFUSE_BATCH_EXPORT_LOCAL_PATH, mirroring the event/media backends, with boot fail-fast validation. The env-schema default iss3so raw deployments keep upstream behavior; the bundled compose stacks (split + standalone) shiplocalso openfuse needs no bucket out of the box.handleBatchExportJob): on the local backend, writes the export to a shared volume and mints a signed, time-limited download token instead of an S3 presigned URL. S3 path unchanged.@langfuse/sharedso the worker mints and the web route verifies with the same secret (NEXTAUTH_SECRET ?? SALT).NEXTAUTH_SECRETis added to shared env and promoted to the shared compose env so web and worker resolve the same value./api/public/batch-exports/[batchExportId]/download: verifies the token, then re-validates thebatchExportrow (project scope,COMPLETEDstatus, expiry) before streaming the file — strictly stronger than a presigned URL. Mirrors the local-media serving precedent.NEXTAUTH_SECRET(so worker-signed tokens verify against web, which signs withNEXTAUTH_SECRETin production) andNEXTAUTH_URLin production (so minted links are reachable, nothttp://localhost:3000); the web runtime requiresLANGFUSE_BATCH_EXPORT_LOCAL_PATH(the download route streams from it). Previously a split deployment could boot a runtime that silently produced unverifiable tokens, dead links, or 500'd every download.langfuse_batch_export_datavolume between web and worker (split + standalone stacks); backend defaults tolocalthere. Dirs pre-created + chowned in the web/worker/standalone images.docs/architecture.md,docs/deployment.md,docs/known-limitations.md,07-deployment.md.Security
The signed token is the presigned-URL equivalent (HMAC, time-limited, scoped to one export); the route additionally re-checks the DB row, so local downloads are not unauthenticated. Secret/URL/path misconfiguration now fails at boot rather than at download time.
Verification
downloadTokenunit tests 8/8, eslint on changed files,docker compose config -qon both compose files — all green.