[cloud] Sync is completely broken, throws 500

[sarink]

Sync has stopped working and now broken my app for new users. It fails with a 500 error and the response Cannot read properties of null (reading 'substring') I don't know what happened. I've reverted to a previously working version and bumped the db version, but it won't sync at all.

[dfahlander]

Thanks for reporting this. I will look into it asap and keep you posted. We upgraded the server this morning

[dfahlander]

@sarink The error seem to be isolated to a few users so far. Can I ask you to trigger a sync request now (for logging purpose)? (if you prefer direct chat on discord, please join https://discord.gg/huhre7MHBF )

[dfahlander]

The issue was triggered by user-specifik data. I've temporarily put your database into staging mode where I've deployed a fix. Can you verify that the issue is resolved? Please confirm the fix and I upgrade production and put your db back.

[sarink]

@dfahlander appears to be working now! Thanks for the quick resolution.

[dfahlander]

Perfect, thanks for verifying! Production will be upgraded by tomorrow and your DB will be back there. The data isn't moved - just the server app.

[sarink]

Sometimes I have had sync fail with "too many requests", I don't exactly know what causes this, but clearing storage (from inspect - application tab) fixes it. Is there any way to detect sync failures and do something like this programmatically?

[dfahlander]

The reason for 429 Too many requests is when hitting a rate-limit, documented at https://dexie.org/docs/cloud/limits
Normally, rate-limit error shouldn't hit sync, since dexie-cloud-addon will debounce the sync calls to avoid it, but it will always repair by itself as the limit is per a certain time window as defined in the link above.

It might be some loop being triggered, for example if your code explicitely calls db.cloud.sync() in a loop instead of relying on the automatic syncing. But it could equally well be some bug in dexie-cloud-addon that makes it retry sync too often.

Make sure you use the latest dexie libs (dexie, dexie-cloud-addon, y-dexie and dexie-react-hooks). If the problem persist, please let me know

[sarink]

I was previously using 4.1-beta, but just upgraded and am now using latest for all dexie libs. We're never calling db.cloud.sync manually. Usually when I've seen this happen it is when a client is a few versions behind, it fires off a zillion requests upon loading, and then gets rate limited. There doesn't appear to be a way to get it un-stuck aside from clearing the storage, which worries me. But I haven't been able to reproduce it consistently.

[sarink]

How would I add a button to destroy the local copy of the indexeddb (thus forcing a clean sync)? The difference between clear and delete are ambiguous to me

[dfahlander]

That sounds bad. This never happens by itself, but could happen if server sends a resync message over the web socket. The reason for that is usually when the end user's set of realms has changed (got member of new realm or lost access to a realm). The client always computes a hash of the ordered list of realms and the server always compares it with the actual set of realms. If there's a diff the client is told over the websocket to do a sync fetch request and then the web socket is reconnected. Maybe this gets into a loop in your case. I haven't experienced this or heard of it before, but this is just one theory to explain what it possibly could be.

If it happens again, let me know, and I could search the log to try get an understanding of the reason.