Fix clone-assets hanging on large spaces (bounded story-fetch concurrency) - #38
Open
alpipego wants to merge 1 commit into
Open
Fix clone-assets hanging on large spaces (bounded story-fetch concurrency)#38alpipego wants to merge 1 commit into
alpipego wants to merge 1 commit into
Conversation
getStories() fetched every story with a single unbounded Promise.all, firing one Management API request per story simultaneously. On a space with a few thousand stories this instantly exceeds the API rate limit and collapses into an endless retry storm, so the migration never reaches the asset-upload step. Fetch stories with async.mapLimit (concurrency = simultaneousUploads, the same knob the upload step already uses) and set an explicit rateLimit on the Management clients. Also bump the cdn/links page size to 100 and add per-story progress output. Verified on a space with 3,941 stories / 1,513 used assets: previously stuck at step 1 with 36k+ rate-limit retries; now completes with zero retries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
clone-assetstool never finishes on spaces with more than a few thousand stories.Problem
getStories()fetches every story with a single unboundedPromise.all, firing one Management API request per story at once:On a large space this instantly exceeds the Management API rate limit and collapses into an endless retry storm, so the migration never reaches the asset-upload step.
Fix
async.mapLimit, reusing the existingsimultaneousUploadsvalue as the concurrency bound (the upload step already works this way).rateLimiton the Management API clients.cdn/linkspage size to 100 and add per-story progress output.No interface change; existing configs behave the same, just without the storm.
Verification
Run against a space with 3,941 stories / 1,513 used assets: previously stuck at step 1 with 36k+ rate-limit retries and 0 assets migrated; after the fix it completed end-to-end with 0 retries, 0 failed assets, all 1,513 assets migrated.