Currently, tractor.dev/toolkit-go/engine/fs provides excellent abstractions for file system operations via fs.FS and MutableFS, with existing backends for local disk, in-memory, and GitHub. However, there isn't a native backend for S3.
Many cloud-native applications, especially those dealing with large quantities of document-like data (like XML, JSON, or media files), rely on S3 as their primary persistent storage. Without a dedicated S3 backend in engine/fs, developers must fall back to direct AWS SDK calls, which bypasses the powerful VFS abstraction that toolkit-go offers. This leads to:
- Increased boilerplate: Repeated S3 SDK calls for common file operations.
- Reduced portability: Code is tied directly to S3 SDK, making it harder to swap storage backends (e.g., for local development with memfs).
- Lack of fs.FS benefits: Cannot leverage io/fs features, or other engine/fs utilities (like UnionFS, WatchFS, or Fuser for S3-backed data).
I propose adding a new S3 backend implementation to tractor.dev/toolkit-go/engine/fs, likely in a new sub-package such as fs/s3fs.
This backend should:
- Implement both the fs.FS and fs.MutableFS interfaces from engine/fs.
- Utilize the AWS SDK for Go v2 for all S3 interactions.
- Support standard S3 authentication mechanisms (IAM roles, environment variables, shared credentials file).
- Handle common S3 operations:
- Open, Stat, ReadDir (simulating directories using prefixes).
- Create, WriteFile, Remove, Rename.
- Mkdir, MkdirAll (creating empty objects with trailing slashes, if desired for console display, or simply ensuring the key prefix for objects).
- Crucially, incorporate optimistic locking for WriteFile / Create operations using S3's ETag or object VersionId to prevent lost updates in concurrent scenarios. This is vital for document-like data (like XML).
- Handle multipart uploads for large files.
- Provide configurable options (e.g., S3 bucket name, region, custom S3 endpoint for testing with localstack/MinIO).
It would allow engine/fs to manage all file-like interactions with S3, ensuring consistency and leveraging toolkit-go's other fs utilities.
Happy to discuss further or potentially contribute to this effort.
Currently, tractor.dev/toolkit-go/engine/fs provides excellent abstractions for file system operations via fs.FS and MutableFS, with existing backends for local disk, in-memory, and GitHub. However, there isn't a native backend for S3.
Many cloud-native applications, especially those dealing with large quantities of document-like data (like XML, JSON, or media files), rely on S3 as their primary persistent storage. Without a dedicated S3 backend in engine/fs, developers must fall back to direct AWS SDK calls, which bypasses the powerful VFS abstraction that toolkit-go offers. This leads to:
I propose adding a new S3 backend implementation to tractor.dev/toolkit-go/engine/fs, likely in a new sub-package such as fs/s3fs.
This backend should:
It would allow engine/fs to manage all file-like interactions with S3, ensuring consistency and leveraging toolkit-go's other fs utilities.
Happy to discuss further or potentially contribute to this effort.