Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft: add ingestdataset function to ingestor app #10

Closed
wants to merge 9 commits into from

Conversation

phwissmann
Copy link
Collaborator

@phwissmann phwissmann commented Sep 5, 2024

Demonstrates the usage of scicat-cli functions to ingest a dataset and adds S3 upload capabilities.

  • Integrate scicat-cli using cli-next branch (contains minimal sciat api/v4 fixes and go module renaming)
  • Add globus config to application config
  • Add S3 upload to ingestion

Todo:

  • Fix Globus integration (GlobusLogin, parameters,...)
  • Remove copied code from scicat-cli (GlobusLogin, createLocalSymlinkCallbackForFileLister, createLocalFilenameFilterCallback)
  • Verify ingestion is correct
  • Align scicat-cli regarding module name and which scicat api version to target

@phwissmann phwissmann force-pushed the 1-create-new-ingestor-application branch from c007a38 to 4913191 Compare September 9, 2024 10:07
Adds ingestion flow with S3 transfer. Relies on sciat-cli with branch cli-next. Stub for globus transfer is implemented taking config from global application config
@phwissmann phwissmann force-pushed the 4-add-ingestdataset-function-to-ingestor-app branch from 08fcd72 to ab983d5 Compare September 9, 2024 10:14
Base automatically changed from 1-create-new-ingestor-application to main September 9, 2024 11:33
@phwissmann phwissmann closed this Sep 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add IngestDataset function to ingestor app
2 participants