fix(runtime): move large session blobs out of entry rows#242
Conversation
mhart
changed the title
mhart
force-pushed
the
mhart/external-session-data
branch
2 times, most recently
from
7d41b3b to
aa2c7b1
Compare
|
no need to rebase / worry about merge conflicts FYI, if we actually decide to ship this well probably do a clean re-implementation. Also you need to handle for Node, not just Cloudflare. Can you look into how hard that would be? |
Yeah this is my habit of amending / rewriting commits coming through – not rebasing to deal with merge conflicts in this case. Still a stickler for a clean commit history! The GH UI has changed over the years to make this look messier than it is though, I agree. Playing with some more generic Node-friendly options as we speak |
Build on the SQL session-entry storage added upstream by moving inline image/blob/document/file payloads out of each entry JSON row. Large payloads are stored in flue_session_blobs and rehydrated on load, preserving session data while avoiding oversized SQLite cells. Fixes withastro#240
Add a generic SessionAttachmentStore option for SQL-backed session blobs so large payloads can be stored outside the database when configured. Cloudflare wires this to an R2 bucket binding, while Node SQLite exposes the same option for custom filesystem, S3-compatible, or application-provided attachment stores. The default remains SQL chunk storage.
mhart
force-pushed
the
mhart/external-session-data
branch
from
aa2c7b1 to
de4617e
Compare
|
Ok, so because this PR introduces chunking of blobs, there's no need for Node users to do anything specific (or Cloudflare users for that matter) – the chunking should handle large attachments anyway. However, you also have the option to use a custom |
Add fileSessionAttachmentStore() for Node SQLite persistence so large session blob attachments can be stored on the local filesystem via the existing SessionAttachmentStore option. The helper writes files under a configured root, rejects path traversal, uses a temporary file plus rename for writes, and keeps SQL as the authoritative attachment manifest.
- Made session blob table rebuilds run inside the Node/Cloudflare transaction wrappers. - Queued cleanup for attempted external uploads when a later upload fails. - Restricted blob ref hydration to actual message content `data`/`blob` fields, so metadata with ref-shaped JSON is preserved. - Tightened `PreparedSessionBlob` typing and removed the double entry-serialization pass. - Added regression tests for upload cleanup, migration rollback, and blob-ref-shaped metadata.
mhart
changed the title
|
Fixed on main by 64ae6e6 with chunked SQLite image persistence plus direct HTTP and SDK image support, so I'm closing this as superseded. Thanks for exploring this space and for the useful reference implementation. Adding a bucket-storage primitive is out of scope for this iteration given the added configuration and lifecycle complexity, but it is definitely worth keeping in mind as attachment-size requirements grow. |
2 participants
Builds on the session-entry storage added in
mainby moving large inline session payloads out of each entry JSON row.Summary
flue_session_blobsinstead of inline influe_session_entries.data.SessionAttachmentStorefor optional external blob storage.FLUE_SESSION_ATTACHMENTSwhen present.flue_session_blobsschemas.Behavior
Without external attachment storage, blobs remain in SQL chunk rows.
On Cloudflare, defining an R2 binding named
FLUE_SESSION_ATTACHMENTSstores large blobs externally in R2. Without that binding, Cloudflare falls back to SQL chunk rows.On Node SQLite, callers can pass a custom
attachmentStore; otherwise Node keeps using SQL chunk rows.Fixes #240