Skip to content

Feat/add services imports#783

Open
KillianCourvoisier wants to merge 39 commits intomasterfrom
feat/addServicesImports
Open

Feat/add services imports#783
KillianCourvoisier wants to merge 39 commits intomasterfrom
feat/addServicesImports

Conversation

@KillianCourvoisier
Copy link
Contributor

@KillianCourvoisier KillianCourvoisier commented Oct 20, 2025

This PR introduces the full integration of the new Data Imports feature into Cozy Settings.

A new entry has been added in the “Data” menu, with three dedicated pages:

  • /imports

    • Lets the user enable or disable the import tools.
  • /imports/run

    • Allows the selection of a provider (only Nextcloud for now — the others are displayed as placeholders).
    • Supports account selection, creation and deletion through a modal. Multiple accounts from different Nextcloud instances can be connected.
    • Lets the user choose the exact remote path (root, directory, or file) and preview the first 10 items to verify the content before launching an import.
    • Displays a real-time progress bar (processed files, total files, and current file), dynamically updating depending on the folder being scanned.
    • Shows the list of errors encountered during the import, with filename and status code, to help understand why specific items were skipped.
    • Allows the user to manually stop the import at any time.
  • /imports/history

    • Currently empty and will be implemented later.

The entire feature is wrapped behind a Cozy feature flag (settings.imports).
If the flag is off, the menu entry and routes are fully hidden.

For now, only Nextcloud accounts are supported. Additional providers such as Google Drive and Dropbox will be added in future updates.

@bundlemon
Copy link

bundlemon bot commented Oct 20, 2025

BundleMon

Files updated (3)
Status Path Size Limits
static/js/cozy.(hash).js
622.69KB (+101.58KB +19.49%) -
static/js/main.(hash).js
147.36KB (+6.11KB +4.33%) -
static/css/main.(hash).css
30.13KB (+262B +0.86%) -
Unchanged files (6)
Status Path Size Limits
static/js/(chunkId).(hash).js
499.5KB -
static/js/lib-react.(hash).js
43.88KB -
static/css/cozy.(hash).css
29.44KB -
static/js/lib-router.(hash).js
21.86KB -
static/svg/export-cozy-mail.(hash).svg
700B -
static/svg/import-cozy.(hash).svg
630B -

Total files change +107.95KB +8.38%

Groups updated (2)
Status Path Size Limits
**/*.js
1.96MB (+109.48KB +5.76%) -
**/*.css
79.36KB (+262B +0.32%) -
Unchanged groups (1)
Status Path Size Limits
**/*.{png,svg,ico}
277KB -

Final result: ✅

View report in BundleMon website ➡️


Current branch size history | Target branch size history

@JF-Cozy
Copy link
Contributor

JF-Cozy commented Oct 20, 2025

@KillianCourvoisier don't hesitate to ping us again when it's ready, I think github doesn't notify us again. You can avoid creating draft if it's not really necessary for testing or specific review of someone for something, because we don't read draft anymore by default

@KillianCourvoisier
Copy link
Contributor Author

@JF-Cozy Yes for sure, no problem, I just used the draft to ensure nobody would try to review it before it is done. But of course when done I'll ping you, thank you for the attention ! :)

Copy link
Contributor

@Crash-- Crash-- left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know the PR is not finished, but at the end, you'll create a PR with 3k lines of code, you should not do that.

We should add code, behind flag, and merge it. Regularly. Because here, it'll be a nightmare to read and if we ask you to change things, it's so easier to change little PR than big ones.

@KillianCourvoisier
Copy link
Contributor Author

Hello @Crash-- , thank you for all the feedback, I'll try my best to do what you asked.

@Crash--
Copy link
Contributor

Crash-- commented Dec 3, 2025

  • Still lot of text used without translation.
  • Still lot of inlined style
  • Still lot of non needed code

But better than ever. Keep pushing!

@zatteo
Copy link
Member

zatteo commented Dec 3, 2025

Can you know before starting the import the total number of files to import? Because currently the progress bar makes no sense because it is increasing the target each time it enters in a new folder.

Screen.Recording.2025-12-03.at.11.08.49.mov

@zatteo
Copy link
Member

zatteo commented Dec 3, 2025

History page makes the Settings bar title disappear

image

@zatteo
Copy link
Member

zatteo commented Dec 3, 2025

Thank you for the PR description that explain very well what is done in the PR @KillianCourvoisier. Do you have more general context about the feature? Like what is the goal of this feature? Why are we doing something different that Nextcloud konnector?

Also what is the purpose of enabling/disabling imports? Are they updated in the background?

@KillianCourvoisier
Copy link
Contributor Author

@Crash-- , thank you for the review.

Still lot of text used without translation.

Yes it true, but I tought it could have waited for the next PR we were mentioning for this kind of correction, do you want me to do it all in here anyway ?

Still lot of inlined style

I'll search on my side, but can you point me a doc or something a could use to get what's expected ? And/or @zatteo maybe ?

Still lot of non needed code

It will be cleaned up


@zatteo , thanks for the feedback too :)

Can you know before starting the import the total number of files [...]

There is no bulk treatment possible with the downstream API (at least for what i know ?) so I technically can, but it will be very greedy as I will have to "pre-scan" the entire folder recursively before launching the actual import.

For example, as it is intended to be a one-shot import tool, most of the users will probably import their files from the root directory directly. For massive instances, this would be hundreds, maybe thousands of requests, just to pre-scan. The solution I found to avoid this was to implement the number of files "toProcess" for each directory I want to import. If you have any suggestion, please don't hesitate as it would be indeed better.

History page makes the Settings bar title disappear

Yes, it is not intended, but i did not work at all on this page yet, it just a "placeholder". I can remove it until i get to work on it if it is a problem 👍

Do you have more general context about the feature? [...]

Yes, sure.
So the Nextcloud konnector only creates "a shortcut" to your Nextcloud instance, you are not really importing (copying) the files from the Nextcloud instance. With this tool, you actually are copying the files from the Nextcloud instance to your Twake Drive. I can actually do a real transfer (as in "completely remove the files from the Nextcloud to put them in Twake Drive"). I don't think I will be asked to ever do this, but it is to make a point. The konnector cannot do that.

For the toggle on import, it was just to act like a "I understand the risks ..." button to allow import tool on your instance. And no, there's no background, everything occurs in the client, if the user reload or go back from the imports/run page during the import, the import is stopped and could be relaunch. No persistence between pages

@Crash--
Copy link
Contributor

Crash-- commented Dec 3, 2025

Why the connector can't do what you explain? You can use the same route & API so we're able to do it also (of course you need to add code to the connector).

But here, what happens here if you close your browser during the import? What happen if your browser lose internet connection? You'll not have this issue with a konnector / service.

@zatteo
Copy link
Member

zatteo commented Dec 3, 2025

About styling, you can either:

Use utility classes from cozy-ui

So instead of having <div style={ display: flex; margin-bottom: 16px;}></div> you can use <div className="u-flex u-mb-1"></div>

You can check every utility classes here and you can also check how it is used in our app codebase:
https://docs.cozy.io/cozy-ui/styleguide/section-utilities.html

Use stylus file

Check how are used .styl files there are plenty in cozy-settings

@KillianCourvoisier
Copy link
Contributor Author

KillianCourvoisier commented Dec 3, 2025

@Crash--

Why the connector can't do what you explain? [...]

Because the konnector was simply designed to create a shared drive, not importing the documents. But it's true, it could be all in there probably .
Only thing is it is not what I was asked to do. They wanted "An import tool from Settings", with no need to connect the konnector, so I have done that ... I can't tell you much about the "why"

But here, what happens here if you close your browser during the import? [...]

I did not push the test to "kill the browser during import", but it stops the import on reload, so I assume it stops the execution when closing the tab totally. I'll try during the new corrections.

@zatteo mucha gracias, I'll take a look on that, it'll help !

@zatteo
Copy link
Member

zatteo commented Dec 3, 2025

@KillianCourvoisier About the progress bar I think it has not real interest as it because it does help to know when it will ends (but I do not say also that you need to precalculate everything to make the progress bar work).

@KillianCourvoisier
Copy link
Contributor Author

@zatteo OK then, i can remove it. But I still think it could be welcome to know how many files are being treated, what do you think ?

Maybe I could simply use a spinner, and let the number of processed files at least ? Let me know

@zatteo
Copy link
Member

zatteo commented Dec 3, 2025

@zatteo OK then, i can remove it. But I still think it could be welcome to know how many files are being treated, what do you think ?

Maybe I could simply use a spinner, and let the number of processed files at least ? Let me know

Good idea you can display the number of files already processed for example. It's clearly better than nothing it will show that something is happening.

@KillianCourvoisier
Copy link
Contributor Author

@Crash-- , @zatteo

I applied corrections to the code according to last review, however regarding i18n and stylus, I just did the NextcloudAccountDialog.jsx file to try to do what has been asked. I wanted to have you check the modifications before going any further. Is it what is expected ? If so, I'll make the changes for every other component 👍

@zatteo
Copy link
Member

zatteo commented Dec 4, 2025

@Crash-- , @zatteo

I applied corrections to the code according to last review, however regarding i18n and stylus, I just did the NextcloudAccountDialog.jsx file to try to do what has been asked. I wanted to have you check the modifications before going any further. Is it what is expected ? If so, I'll make the changes for every other component 👍

It's exactly this for styling 👍

@KillianCourvoisier
Copy link
Contributor Author

Ok, all i18n translation and usage are done too, I hope everything is fine now. Lemme know

Copy link
Contributor

@Crash-- Crash-- left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still super complicated to understand.

If I have to maintain that by myself, I'm not very confident.

err.status = 404
err.path = clean
throw err
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly:

  • Let's say I call probePath with /foo/bar/
  • I'll call listRemote with /foo
  • Then I loop on all the result on /foo to see if bar really exist?
  • And if not exist, we can throw an error with `weird characters name' . (wtf?)
  • And if exist, then we check the type, and if it's a directory, then we finally list the files of the directory.

So:

  • If I have a lot of folder / files in /foo what happen? Do we've pagination on this route? Can it be very slow? Can't we already know the type of the entry?
  • Can we really call this route with a file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current probePath behavior follows what I could find for NextcloudFilesCollection.

The collection only exposes a directory-level listing (find() lists children of a given parentPath), to check a path like /foo/bar I need to do as follows :

  • Listing the parent directory (/foo)
  • Locating the entry named bar
  • Fetching its children only if it is a directory

I've done this to avoid introducing custom remote calls, as requested

On performance: the endpoint used by NextcloudFilesCollection seems to not support pagination, so the cost is the same whether the lookup happens inside our code or directly through the collection.

@KillianCourvoisier
Copy link
Contributor Author

Hello @Crash-- , I tried to address everything you pointed out on the last review, here is what I've changed :

  • Removed custom fetch usage in destinationService, now relying on ensureDirectoryExists.
  • Renamed docs to nextcloudAccounts.
  • Simplified readError (no async, directly surfaces Cozy error payload).
  • Fixed silent abort in import flow (proper abort flag + limiter reject).
  • Cleaned up inconsistent try/catch logic in remoteService.
  • Removed “weird characters” handling, standardized error messaging.
  • Dropped build404Reason; 404s now use a single normalized message: “not found on remote provider”.

I think it resolve most of the problem you were mentioning,

I still have stuff to do like a health check of the account upon creation, to hide the listing/importing section if the account is not OK or change the progress bar to a spinner + processedFiles number to avoid confusion when the bar is dropping back on each new scanned folder but I was thinking it could have been for another PR.

Let me know what you think 👍

@Crash--
Copy link
Contributor

Crash-- commented Dec 9, 2025

I still think this code should belongs to nextcloud connector & not be part of this app.

With connectors, we already have runs history, we have "stats" (files imported), we have "background fetching". I think it's a terrible idea to keep all the logic here.

Maybe we can keep the modal to create the account if we don't want to use harvest... but that's it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants