How to avoid HTTP timeouts when syncing very large batches (need chunked/incremental filtering strategy?) #1345
Unanswered
federicoMoroni-git
asked this question in
Q&A
Replies: 1 comment
-
|
We dealt with this by syncing tables one at a time in the order of the relational hierarchy. We then do a final sync of all tables which doesn't sync any data but sets a baseline for future syncs. This worked most of the time but there was the occasional table for some large customers that would still timeout. So we moved away from Azure. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I’m running into a recurring problem when syncing large tables over HTTP with Dotmim.Sync.
Whenever there are many thousands of changes, the client starts the sync, but the HTTP request times out before the whole dataset can be transferred/applied.
After ~13 minutes after the beginning of the sync method, the HTTP request fails with:
Sync failed with error:
<html><head><title>500 - The request timed out.</title></head><body><font color ="#aa0000"><h2>500 - The request timed out.</h2></font>The web server failed to respond within the specified time.</body></html>I`ve already tried to increase the timout limit of my server but as far as I Understood for Azure hosted ASP core apps there is a limit which should be around 4 minutes.
Server: Azure hosted ASP.NET Core app with NpgsqlSyncProvider (PostgreSQL backend).
Client: SQLite with SqliteSyncProvider.
What I tried so far:
Server-side (web.config in wwwroot of ASP.NET Core app)
<aspNetCore processPath="dotnet" arguments=".\WebAppWithAD.dll" hostingModel="inprocess" requestTimeout="01:00:00" /> <!-- 1 hour --> <system.web> <httpRuntime executionTimeout="7200" /> <!-- 2 hours --> </system.web>Client-side (Console app with SQLite)
var orchestrator = new WebRemoteOrchestrator("http://my.syncapi.com:88/Sync"); orchestrator.HttpClient.Timeout = TimeSpan.FromMinutes(20);Increased DbCommandTimeout = 6000 in SyncOptions.
My Plan B would be to use filters, like a rolling filter strategy to divide the upload in chinks (for example based on timestamps) and upload them incrementally, but so far I haven
t succeed in developing such a solution any help here? PS: I cant use the TCP sync in this context because it`s not reliable with poor connectivity environmentBeta Was this translation helpful? Give feedback.
All reactions