Skip to content

feat: replace afero-s3 with minimal in-tree S3 implementation#2348

Open
maggch97 wants to merge 3 commits intogo-vikunja:mainfrom
maggch97:feat/replace-afero-s3
Open

feat: replace afero-s3 with minimal in-tree S3 implementation#2348
maggch97 wants to merge 3 commits intogo-vikunja:mainfrom
maggch97:feat/replace-afero-s3

Conversation

@maggch97
Copy link
Contributor

@maggch97 maggch97 commented Mar 3, 2026

Summary

Replaces the third-party fclairamb/afero-s3 library (and its temporary fork) with a minimal in-tree afero.Fs implementation (~200 lines) that only supports the three S3 operations Vikunja actually uses:

  • OpenHeadObject (existence check) + lazy GetObject on first Read
  • RemoveHeadObject + DeleteObject
  • StatHeadObject

All other afero.Fs/afero.File methods return ErrS3NotSupported since they are never called in the S3 code path (writes already use direct PutObject).

Changes

  1. pkg/files/s3fs.go (new) — Minimal S3 afero.Fs + afero.File + os.FileInfo implementation
  2. pkg/files/filehandling.go — Replace aferos3.NewFsFromClient() with newS3Fs()
  3. pkg/routes/api/v1/user_export.go — Add S3 branch for export download (io.Copy instead of http.ServeContent which requires io.ReadSeeker)
  4. go.mod / go.sum — Remove fclairamb/afero-s3 dependency and replace directive

Why

  • Removes dependency on a library that caused a severe S3 read performance regression (fix(deps): use forked afero-s3 to fix S3 read performance regression #2313)
  • Removes the temporary fork replace directive
  • The full afero-s3 library implements ~400 lines of S3 operations we never use (Create, Mkdir, Rename, Chmod, Write streams, Readdir, etc.)
  • Our implementation is small enough to maintain in-tree with minimal effort

Closes #2347

maggch97 and others added 2 commits March 3, 2026 21:28
Replace the third-party afero-s3 library (and its temporary fork) with
a minimal in-tree afero.Fs implementation (~200 lines) that only supports
the three S3 operations Vikunja actually uses: Open (GetObject), Remove
(DeleteObject), and Stat (HeadObject).

The s3File implementation lazily opens a GetObject stream on first Read
and supports Seek by closing and re-opening the stream from the new offset,
matching the behavior of the fixed afero-s3 fork.

All other afero.Fs/File methods return ErrS3NotSupported since they are
never called in the S3 code path (writes already use direct PutObject).

This removes the fclairamb/afero-s3 dependency and the temporary replace
directive in go.mod entirely.

Closes go-vikunja#2347

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
http.ServeContent requires io.ReadSeeker which S3 files don't support.
Add S3 branch using io.Copy with explicit headers, matching the pattern
already used in task attachment downloads.

Refs: go-vikunja#2347

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
)

// s3Fs is a minimal afero.Fs implementation backed by S3.
// It only supports Open (read), Remove, and Stat — the operations
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No create or write? how exactly does that work then to create upload attachments?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's handled by

func writeToStorage(path string, content io.ReadSeeker, size uint64) error {

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, but if we're bypassing the s3 fs there already, what advantage do we have to reimplement it? (albeit partially)

@maggch97 maggch97 requested a review from kolaente March 3, 2026 14:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evaluate replacing afero-s3 with direct S3 SDK usage

2 participants