Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Go prototype for s3 upload #51

Open
phwissmann opened this issue May 2, 2024 · 1 comment
Open

[Feature] Go prototype for s3 upload #51

phwissmann opened this issue May 2, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@phwissmann
Copy link
Collaborator

Description
As a user I want to schedule uploads of my datasets by selecting a folder. Uploads should happen in parallel (or sequence) for different datasets. The upload should pick the next dataset(s) in the queue to be uploaded. The path within the s3 bucket can be specified.

Solution proposals
Goroutines provide a good way to schedule tasks by using pipelines with Bounded parallelism.
https://go.dev/blog/pipelines

Definition of Done
Multiple folders can be selected by a frontend and the contents will be uploaded to a minio bucket.

@phwissmann phwissmann added the enhancement New feature or request label May 2, 2024
@phwissmann phwissmann added this to the v1.0-alpha milestone May 2, 2024
@phwissmann phwissmann self-assigned this May 2, 2024
@phwissmann
Copy link
Collaborator Author

Test application implemented:

  • Multiple parallel uploads possible
  • Upload rate: between 2Gbps and 3.6 Gbps, with md5 checksum
  • Runs on Win (transfer computer) and Linux, as expected.

@phwissmann phwissmann changed the title [Feature] Go module for s3 upload [Feature] Go prototype for s3 upload Jun 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant