[v4] Migrations run fetches max 25 stories (does not fetch all matching stories)

[edvinasjurele]

Current behavior:

✔ Fetched 25 stories with related content (filtered by component "picture").
✔ Starting to process 25 stories with 1 migrations...

NOTE: we have 7059 stories containing picture block, while only 25 were fetched by CLI

Expected behavior:

CLI fetches all matching stories (even thousands) from Storyblok and successfully applies migration on them.

Steps to reproduce:

1. Have space with more than 25 items matching search by component (f.e. picture)
2. Run migrations run picture --space <SPACE_ID> --verbose
3. Observe terminal logs

Related code:

I do find this fetchStories utility we have in one of our projects. Note that it is for fetching Management API data from CMS, but I am sure the same approach of reading total and perPage from the first RESPONSE can be applied to solving this very task too:

import StoryblokClient from 'storyblok-js-client';

export const fetchStories = async ({ spaceId, params }) => {
    const stories = [];
    const Storyblok = new StoryblokClient({
        oauthToken: process.env.STORYBLOK_TOKEN,
        region: 'us',
        cache: {
            type: 'none'
        },
    });

    await Storyblok.flushCache();

    const response = await Storyblok.get(`spaces/${spaceId}/stories`, {
        ...params,
        per_page: 100,
    });
    const { perPage, total } = response;
    const pageCount = Math.ceil(total / perPage);

    stories.push(...response.data.stories);

    for (let page = 2; page <= pageCount; page++) {
        const response = await Storyblok.get(`spaces/${spaceId}/stories`, {
            ...params,
            version: 'published',
            per_page: 100,
            page,
        });
        stories.push(...response.data.stories);
    }

    return stories;
};

Additionally, since fetching 7000 pages takes some time (approax 700ms for 70 requests = ~49 seconds), it would also be nice to do some console logging, to see the fetching progress - print out how many matches found in total, how many pages, or which page is being fetched.

Other information: