Skip to content

Fix clone-assets hanging on large spaces (bounded story-fetch concurrency) - #38

Open
alpipego wants to merge 1 commit into
storyblok:mainfrom
alpipego:fix/large-space-story-fetch-concurrency
Open

Fix clone-assets hanging on large spaces (bounded story-fetch concurrency)#38
alpipego wants to merge 1 commit into
storyblok:mainfrom
alpipego:fix/large-space-story-fetch-concurrency

Conversation

@alpipego

Copy link
Copy Markdown

The clone-assets tool never finishes on spaces with more than a few thousand stories.

Problem

getStories() fetches every story with a single unbounded Promise.all, firing one Management API request per story at once:

const storiesResponsesManagement = await Promise.all(
  links.map((link) =>
    this.targetMapiClient.get(`spaces/${this.targetSpaceId}/stories/${link.id}`)
  )
);

On a large space this instantly exceeds the Management API rate limit and collapses into an endless retry storm, so the migration never reaches the asset-upload step.

Fix

  • Fetch stories with async.mapLimit, reusing the existing simultaneousUploads value as the concurrency bound (the upload step already works this way).
  • Set an explicit rateLimit on the Management API clients.
  • Bump the cdn/links page size to 100 and add per-story progress output.

No interface change; existing configs behave the same, just without the storm.

Verification

Run against a space with 3,941 stories / 1,513 used assets: previously stuck at step 1 with 36k+ rate-limit retries and 0 assets migrated; after the fix it completed end-to-end with 0 retries, 0 failed assets, all 1,513 assets migrated.

getStories() fetched every story with a single unbounded Promise.all,
firing one Management API request per story simultaneously. On a space
with a few thousand stories this instantly exceeds the API rate limit
and collapses into an endless retry storm, so the migration never
reaches the asset-upload step.

Fetch stories with async.mapLimit (concurrency = simultaneousUploads,
the same knob the upload step already uses) and set an explicit
rateLimit on the Management clients. Also bump the cdn/links page size
to 100 and add per-story progress output.

Verified on a space with 3,941 stories / 1,513 used assets: previously
stuck at step 1 with 36k+ rate-limit retries; now completes with zero
retries.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant