Problem with sync_list

[lennartjakob]

Hey,

i wanted to use the discussion feature to ask about a problem i am facing with the sync_list and understanding how it works.
I am trying to sync specific folders of a sharepoint library and am facing long waiting times
My sync_list looks like this:

# Exclusions
!/1️⃣ Verwaltung/
!/2️⃣ Seminare/
!/4️⃣ Dienstformen/
!/Email attachments/
!/General/
# Inclusions
/3️⃣ iFWD/09 Spenden/*
/weltwärts*/*Einzelordner*/*_Spenden_*.xlsx
/weltwärts*/*Einzelordner*/*_Spenden_*.pdf

We shall not talk about the emojis in the names 😭

Now to my problem: When I launch a sync with --download-only, the client checks all folders of the sharepoint site (correctly ignores according to the list though), which takes roughly 15 minutes, because there are like 20000 Folders. If i have to run the sync multiple times in a day i get rate-limited by the Graph API.
I run everything verbose, so i can see that there is output like
Skipping path - excluded by sync_list config: ./2️⃣ Seminare/XXX/YYY/ZZZ/111/222/333

Now for my questions
Is this expected behavior?
From this thread i gather that it is
And if not, how can i fix my configuration to actually exclude the mentioned folders?

Does the --download-only option play into this?

For later syncs it works just fine, but my client only tries syncing once a week, because it is part of a cron job. Problem is, that every other week it asks for a resync, which then takes long again.

Does anyone in the community have an idea how to fix this? Should i maybe create a live-synced client, to combat the 1 week time-out issue?

[abraunegg]

@lennartjakob

Your description suggests a misunderstanding of how the client operates, particularly with respect to sync_list.

The OneDrive Client for Linux performs client-side filtering, not server-side filtering. This means the client must still enumerate the entire remote directory structure to determine what exists, and then it applies the sync_list rules locally to decide what should actually be downloaded or ignored. In other words, sync_list does not reduce what the Microsoft API returns, it only controls what the client processes locally.

Because of this design, the initial scan or a --resync will still require a full online enumeration regardless of your sync_list configuration.

Regarding the behaviour you are seeing:

“every other week it asks for a resync”

That is not expected behaviour in normal operation. The client only requests a --resync when it detects that the local sync state is no longer valid or the respective local configuration has changed. A --resync explicitly discards the stored client state and performs a full rebuild of the sync database.

A resync prompt is typically triggered when these configuration parameters or the runtime environment change, for example:

• Different configuration file being used
• Different sync_dir
• Different account / drive_id
• Changes to filtering options (skip_file, skip_dir, etc.)
• Running the client under a different user or environment

These scenarios are documented here:
https://github.com/abraunegg/onedrive/blob/master/docs/usage.md#performing-a---resync

Given that you are running the client via a weekly cron job, the most likely explanation is that the job is executing in a different runtime context than you expect. Cron runs with a minimal environment, so common issues include:

• The job running as a different user
• $HOME pointing to a different location
• A different configuration directory being used
• Different command options being supplied
• Multiple configurations being used unintentionally

When this happens, the client sees a different configuration or state database, and it correctly determines that a --resync is required.

To diagnose this, please provide:

• The exact cron job entry
• The full command being executed
• The user account that cron runs the job as
• The path to your configuration file

The first ~400 lines of the client log output when the cronjob is running. Please enable debug mode (--verbose --verbose) to ensure client output is at a level that is meaningful to assist with this issue.

Once that information is available it should be possible to identify why the client believes the configuration has changed and is requesting a resync.

[abraunegg]

@lennartjakob

Further to the above, please review the Server Side Filtering documentation which explains why Client Side Filtering is the only option, and why enumeration of your online state is returned, only to be excluded by the filtering rules

Please read: https://github.com/abraunegg/onedrive/blob/master/docs/server-side-filtering-limitations.md

[lennartjakob]

@abraunegg

Thank you for your ever fast responses and explanation.

I was using nginx to trigger the sync and onedrive running inside a docker container. To the best of my knowledge everything was run as root and the microsoft account was always the same.
It has been a couple of months since my last test of having it run autonomous, so i currently don't possess any of the logs where it failed, because i just set aside some weekly time to run everything manually, which i currently want to change again.

I will do the steps you just pointed me to and try to set up the weekly runner again, keeping tabs on log output if anything fails.

I will get back here in one or two weeks, if either everything works, or i have interesting logs to present.

What i do remember from the last time that i tried having a configuration persist over weeks is, that if i didn't run a sync in over a week, the error causing me to be prompted for resync was, that my onedrive account seemed to no longer be authenticated.

Again, many thanks for the great reply and i will keep you posted.

[abraunegg]

@lennartjakob

I was using nginx to trigger the sync and onedrive running inside a docker container. To the best of my knowledge everything was run as root and the microsoft account was always the same.
.......
What i do remember from the last time that i tried having a configuration persist over weeks is, that if i didn't run a sync in over a week, the error causing me to be prompted for resync was, that my onedrive account seemed to no longer be authenticated.

So there are 2 potential things for you to investigate:

1. You change your password for your account weekly or more frequently .... that causes a --reauth not a --resync requirement.
2. This suspiciously sounds similar to Bug: Auth is lost after container restart #3492 ... have you followed the Docker deployment to-the-letter and not made some assumption on the /onedrive/conf volume - thus when the Docker container is restarted - there is no data that is persisted (such as the 'refresh_token' .. which would then make the Docker container no longer authenticated .........

[lennartjakob]

I thought of those as well, but i have ensured that the script only runs once all persistent data is present and the password also didn't change, ever.
I will go over the documentation once more when i get the chance to set everything up again and keep you in the loop.