Draft: add ingestdataset function to ingestor app

.github/workflows/ci.yaml

@@ -0,0 +1,56 @@
+# This workflow will build a golang project
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-go
+
+name: Build and Test Workflow
+
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
+
+jobs:
+  lint:
+    name: Lint files
+    runs-on: 'ubuntu-latest'
+    steps:
+      - uses: actions/[email protected]
+      - uses: actions/setup-go@v2
+        with:
+          go-version: '1.21'
+      - name: golangci-lint
+        uses: golangci/[email protected]
+        with:
+          version: v1.60.3
+
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: Set up Go
+      uses: actions/setup-go@v4
+      with:
+        go-version: '1.21'
+
+    - name: Test internal
+      run: go test -C internal/* -v
+
+    - name: Build Service
+      run: go build -C cmd/openem-ingestor-service/ -v
+
+    - name: Test Service
+      run: go test -C cmd/openem-ingestor-service/ -v
+
+    - name: Install Wails dependencies
+      run: sudo apt-get install -qq -y libwebkit2gtk-4.0-dev libgtk-3-dev npm pkg-config
+
+    - name: Set up Wails
+      run: go install github.com/wailsapp/wails/v2/cmd/wails@latest
+
+    - name: Build Desktop App
+      working-directory: cmd/openem-ingestor-app/
+      run: wails build
+
+    - name: Test Desktop App
+      run: go test -C cmd/openem-ingestor-app/ -v

.gitignore

@@ -23,3 +23,5 @@ go.work.sum
 
 # env file
 .env
+
+**/build/**

.vscode/launch.json

@@ -0,0 +1,25 @@
+{
+  "version": "0.2.0",
+  "configurations": [
+    {
+      "name": "Wails: Debug Desktop App",
+      "type": "go",
+      "request": "launch",
+      "mode": "exec",
+      "program": "${workspaceFolder}/cmd/openem-ingestor-app/build/bin/openem-ingestor-app",
+      "preLaunchTask": "build app",
+      "cwd": "${workspaceFolder}/cmd/openem-ingestor-app/build/bin",
+      "env": {}
+    },
+    {
+      "name": "Wails: Debug Service",
+      "type": "go",
+      "request": "launch",
+      "mode": "exec",
+      "program": "${workspaceFolder}/cmd/openem-ingestor-service/build/bin/openem-ingestor-service",
+      "preLaunchTask": "build service",
+      "cwd": "${workspaceFolder}/cmd/openem-ingestor-service/build/bin",
+      "env": {}
+    }
+  ]
+}

.vscode/tasks.json

@@ -0,0 +1,76 @@
+{
+  "version": "2.0.0",
+  "tasks": [
+    {
+      "label": "copy app config",
+      "type": "shell",
+      "command": "cp",
+      "args": [
+        "${workspaceFolder}/configs/openem-ingestor-config.yaml",
+        "${workspaceFolder}/cmd/openem-ingestor-app/build/bin/"
+      ]
+    },
+    {
+      "label": "copy service config",
+      "type": "shell",
+      "command": "cp",
+      "args": [
+        "${workspaceFolder}/service/openem-ingestor-config.yaml",
+        "${workspaceFolder}/cmd/openem-ingestor-service/build/bin/"
+      ]
+    },
+    {
+      "label": "build desktop app",
+      "type": "shell",
+      "options": {
+        "cwd": "${workspaceFolder}/cmd/openem-ingestor-app"
+      },
+      "command": "wails",
+      "args": [
+        "build",
+        "-debug"
+      ]
+    },
+    {
+      "label": "build service app",
+      "type": "shell",
+      "options": {
+        "cwd": "${workspaceFolder}/cmd/openem-ingestor-service"
+      },
+      "command": "go",
+      "args": [
+        "build",
+        "-tags",
+        "dev",
+        "-gcflags",
+        "all=-N -l",
+        "-o",
+        "build/bin/openem-ingestor-service"
+      ]
+    },
+    {
+      "label": "build app",
+      "group": {
+        "kind": "build",
+        "isDefault": true
+      },
+      "dependsOn": [
+        "build desktop app",
+        // Enable this to use local config during debugging
+        // "copy app config"
+      ]
+    },
+    {
+      "label": "build service",
+      "group": {
+        "kind": "build",
+        "isDefault": false
+      },
+      "dependsOn": [
+        "build service app",
+        // Enable this to use local config during debugging
+        // "copy service config"
+      ]
+    },
+  ]
+}

README.md

@@ -1,2 +1,39 @@
-# Ingestor
-OpenEM Data Network Ingestor service and app
+# OpenEm Data Network Ingestor
+
+This repository provides an ingestion app and service for dataset transfer and metadata registration in a catalog. It targets [Scicat](https://scicatproject.github.io) as dataset catalog.
+
+Data can be transfered via [Globus](https://www.globus.org) or via S3 to a compatible endpoint.
+
+There are two entrypoints, i.e. applications: a [desktop app](./cmd/openem-ingestor-app/) providing a minimal UI and a headless [service](./cmd/openem-ingestor-service/). Both provide a REST API in order to interact with each. Common functionality is extracted in the [core](./internal/) package.
+
+## Core
+
+The core package contains shared functionality between the desktop app and the service. It makes use of the [scicat-cli tools](https://github.com/paulscherrerinstitute/scicat-cli/tree/main) for interactions with Scicat. Two APIs are provided; a REST API for it interact with it as a service, and a Go API to interact with it within the same application.
+
+## Desktop App
+
+The desktop app is based on [wails.io](https://wails.io) which provides bindings between Go and typscript in order to write portable frontends in various web frameworks. Svelte was chosen in this case.
+
+For development wails provides hot reload capabilities:
+
+```bash
+../Ingestor/desktop-app> wails dev
+```
+
+And a build command to build frontend and backend into a single executable:
+
+```bash
+../Ingestor/desktop-app> wails build
+```
+
+see [wails.io](https://wails.io) for details.
+
+## Service
+
+```bash
+../Ingestor/desktop-app> go build
+```
+
+## Debugging
+
+[launch.json](.vscode/launch.json) and [task.json](.vscode/tasks.json) are provided to define debug targets for VS Code.

cmd/openem-ingestor-app/app.go

@@ -0,0 +1,94 @@
+package main
+
+import (
+	"context"
+
+	core "github.com/SwissOpenEM/Ingestor/internal/core"
+
+	"github.com/google/uuid"
+	"github.com/wailsapp/wails/v2/pkg/runtime"
+)
+
+type WailsNotifier struct {
+	AppContext context.Context
+}
+
+func (w *WailsNotifier) OnTaskScheduled(id uuid.UUID) {
+	runtime.EventsEmit(w.AppContext, "upload-scheduled", id)
+}
+func (w *WailsNotifier) OnTaskCanceled(id uuid.UUID) {
+	runtime.EventsEmit(w.AppContext, "upload-canceled", id)
+}
+func (w *WailsNotifier) OnTaskRemoved(id uuid.UUID) {
+	runtime.EventsEmit(w.AppContext, "folder-removed", id)
+}
+func (w *WailsNotifier) OnTaskFailed(id uuid.UUID, err error) {
+	runtime.EventsEmit(w.AppContext, "upload-failed", id, err.Error())
+}
+func (w *WailsNotifier) OnTaskCompleted(id uuid.UUID, seconds_elapsed int) {
+	runtime.EventsEmit(w.AppContext, "upload-completed", id, seconds_elapsed)
+}
+func (w *WailsNotifier) OnTaskProgress(id uuid.UUID, current_file int, total_files int, elapsed_seconds int) {
+	runtime.EventsEmit(w.AppContext, "progress-update", id, current_file, total_files, elapsed_seconds)
+}
+
+// App struct
+type App struct {
+	ctx       context.Context
+	taskqueue core.TaskQueue
+	config    core.Config
+}
+
+// NewApp creates a new App application struct
+func NewApp(config core.Config) *App {
+	return &App{config: config}
+}
+
+// Show prompt before closing the app
+func (b *App) beforeClose(ctx context.Context) (prevent bool) {
+	dialog, err := runtime.MessageDialog(ctx, runtime.MessageDialogOptions{
+		Type:    runtime.QuestionDialog,
+		Title:   "Quit?",
+		Message: "Are you sure you want to quit? This will stop all pending downloads.",
+	})
+
+	if err != nil {
+		return false
+	}
+	return dialog != "Yes"
+}
+
+// startup is called when the app starts. The context is saved
+// so we can call the runtime methods
+func (a *App) startup(ctx context.Context) {
+	a.ctx = ctx
+	a.taskqueue = core.TaskQueue{Config: a.config,
+		AppContext: a.ctx,
+		Notifier:   &WailsNotifier{AppContext: a.ctx},
+	}
+	a.taskqueue.Startup()
+}
+
+func (a *App) SelectFolder() {
+	folder, err := core.SelectFolder(a.ctx)
+	if err != nil {
+		return
+	}
+
+	err = a.taskqueue.CreateTask(folder)
+	if err != nil {
+		return
+	}
+}
+
+func (a *App) CancelTask(id uuid.UUID) {
+	a.taskqueue.CancelTask(id)
+}
+func (a *App) RemoveTask(id uuid.UUID) {
+	a.taskqueue.RemoveTask(id)
+}
+
+func (a *App) ScheduleTask(id uuid.UUID) {
+
+	a.taskqueue.ScheduleTask(id)
+}

cmd/openem-ingestor-app/frontend/.gitignore

@@ -0,0 +1,3 @@
+node_modules/**
+dist/**
+!dist

cmd/openem-ingestor-app/frontend/.vscode/extensions.json

@@ -0,0 +1,5 @@
+{
+  "recommendations": [
+    "svelte.svelte-vscode"
+  ]
+}

cmd/openem-ingestor-app/frontend/README.md

@@ -0,0 +1,65 @@
+# Svelte + TS + Vite
+
+This template should help get you started developing with Svelte and TypeScript in Vite.
+
+## Recommended IDE Setup
+
+[VS Code](https://code.visualstudio.com/)
+
++ [Svelte](https://marketplace.visualstudio.com/items?itemName=svelte.svelte-vscode).
+
+## Need an official Svelte framework?
+
+Check out [SvelteKit](https://github.com/sveltejs/kit#readme), which is also powered by Vite. Deploy anywhere with its
+serverless-first approach and adapt to various platforms, with out of the box support for TypeScript, SCSS, and Less,
+and easily-added support for mdsvex, GraphQL, PostCSS, Tailwind CSS, and more.
+
+## Technical considerations
+
+**Why use this over SvelteKit?**
+
+- It brings its own routing solution which might not be preferable for some users.
+- It is first and foremost a framework that just happens to use Vite under the hood, not a Vite app.
+  `vite dev` and `vite build` wouldn't work in a SvelteKit environment, for example.
+
+This template contains as little as possible to get started with Vite + TypeScript + Svelte, while taking into account
+the developer experience with regards to HMR and intellisense. It demonstrates capabilities on par with the
+other `create-vite` templates and is a good starting point for beginners dipping their toes into a Vite + Svelte
+project.
+
+Should you later need the extended capabilities and extensibility provided by SvelteKit, the template has been
+structured similarly to SvelteKit so that it is easy to migrate.
+
+**Why `global.d.ts` instead of `compilerOptions.types` inside `jsconfig.json` or `tsconfig.json`?**
+
+Setting `compilerOptions.types` shuts out all other types not explicitly listed in the configuration. Using triple-slash
+references keeps the default TypeScript setting of accepting type information from the entire workspace, while also
+adding `svelte` and `vite/client` type information.
+
+**Why include `.vscode/extensions.json`?**
+
+Other templates indirectly recommend extensions via the README, but this file allows VS Code to prompt the user to
+install the recommended extension upon opening the project.
+
+**Why enable `allowJs` in the TS template?**
+
+While `allowJs: false` would indeed prevent the use of `.js` files in the project, it does not prevent the use of
+JavaScript syntax in `.svelte` files. In addition, it would force `checkJs: false`, bringing the worst of both worlds:
+not being able to guarantee the entire codebase is TypeScript, and also having worse typechecking for the existing
+JavaScript. In addition, there are valid use cases in which a mixed codebase may be relevant.
+
+**Why is HMR not preserving my local component state?**
+
+HMR state preservation comes with a number of gotchas! It has been disabled by default in both `svelte-hmr`
+and `@sveltejs/vite-plugin-svelte` due to its often surprising behavior. You can read the
+details [here](https://github.com/rixo/svelte-hmr#svelte-hmr).
+
+If you have state that's important to retain within a component, consider creating an external store which would not be
+replaced by HMR.
+
+```ts
+// store.ts
+// An extremely simple external store
+import { writable } from 'svelte/store'
+export default writable(0)
+```

cmd/openem-ingestor-app/frontend/index.html

@@ -0,0 +1,12 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8"/>
+    <meta content="width=device-width, initial-scale=1.0" name="viewport"/>
+    <title>openem-ingestor</title>
+</head>
+<body>
+<div id="app"></div>
+<script src="./src/main.ts" type="module"></script>
+</body>
+</html>