Skip to content

Add S3 File Provider #2164

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 7 commits into
base: main
Choose a base branch
from
Draft

Add S3 File Provider #2164

wants to merge 7 commits into from

Conversation

NolanTrem
Copy link
Collaborator

@NolanTrem NolanTrem commented Apr 30, 2025

WIP: Will be released with v3.6.0

Adds support for S3 as a file provider as a configurable alternative to the existing Postgres Large Objects table which is used to store files.

Outstanding issues/work:

  • Test with additional S3 providers
  • Documentation, examples
  • S3 ingestion via SDK
  • MinIO Docker
  • Supabase is not working with certain file types.

MinIO:
Screenshot 2025-05-02 at 11 03 26 AM

Supabase:
Screenshot 2025-05-01 at 2 24 00 PM

See: supabase/storage#639
The following errors are raised when using Supabase buckets, but not other S3 compatible buckets.

File Type Error Reason
HEIC Invalid file structure: No 'ftyp' box at the beginning of the file
JPEG MIME type validation error: Only certain image types are supported
MSG Invalid file structure: Not an OLE2 structured storage file
PPT Variable access error: Cannot access local variable 'ole'
TIFF MIME type validation error: Only certain image types are supported
P7S Invalid S/MIME structure: Does not contain a valid signed message
PNG MIME type validation error: Only certain image types are supported
JPG MIME type validation error: Only certain image types are supported
JSON Invalid JSON format: Extra data detected in line 2 column 1
XLS Corrupt or unsupported format: Expected BOF record not found
DOC Variable access error: Cannot access local variable 'ole'

List of supported file storage providers.
"""
return [
"postgres",

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A shot in the dark here, but should this be "supabase"?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'll keep this to s3, since it should support any s3 compatible solution (not just Supabase).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants