[DEV-2583] Add optimized GitBook sync workflow and metadata synchronization script #1711

tommaso1 · 2025-09-30T13:11:02Z

List of Changes

Optimize GitBook sync workflow to reduce API calls and improve performance

Add unified metadata generation script that reduces Strapi API calls from 8 to 5
Implement S3 operations caching to avoid duplicate file downloads
Parallelize all Strapi fetches using Promise.all()
Create optimized workflow that uses the unified script
Maintain backward compatibility with existing workflow

Motivation and Context

How Has This Been Tested?

Frist go to the gitbook docs folder
cd packages/gitbook-docs
compile
npm run compile
set env variables

export GENERATE_URL_METADATA=false
export METADATA_TYPE=guides
export S3_BUCKET_NAME=devportal-d-website-static-content
export ENVIRONMENT=dev

or 

export GENERATE_URL_METADATA=false
export METADATA_TYPE=all
export S3_BUCKET_NAME=devportal-d-website-static-content
export ENVIRONMENT=dev

run the script
npm run sync-all-metadata

Screenshots (if appropriate):

Types of changes

Chore (nothing changes by a user perspective)
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

My change requires a change to the documentation.
I have updated the documentation accordingly.

changeset-bot · 2025-09-30T13:11:08Z

🦋 Changeset detected

Latest commit: c20a346

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
gitbook-docs	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

marcobottaro · 2025-10-07T14:08:09Z

packages/gitbook-docs/src/scripts/syncAllMetadata.ts

+  guides: StrapiGuide[];
+  solutions: StrapiSolution[];
+  releaseNotes: StrapiReleaseNote[];
+  guidesResponse?: any;


Maybe you should find a more explicit name for *Response attributes

marcobottaro · 2025-10-07T14:08:49Z

packages/gitbook-docs/src/scripts/syncAllMetadata.ts

+  }[];
+}
+
+let s3Client: ReturnType<typeof makeS3Client> | undefined;


Why don't you use S3Client?

Suggested change

let s3Client: ReturnType<typeof makeS3Client> | undefined;

let s3Client: S3Client | undefined;

marcobottaro · 2025-10-07T14:09:03Z

packages/gitbook-docs/src/scripts/syncAllMetadata.ts

+
+let s3Client: ReturnType<typeof makeS3Client> | undefined;
+
+function getS3Client(): ReturnType<typeof makeS3Client> {


marcobottaro · 2025-10-07T14:21:27Z

packages/gitbook-docs/src/scripts/syncAllMetadata.ts

+      'api/release-notes/?populate[bannerLinks][populate][0]=icon&populate[product][populate][0]=logo&populate[product][populate][1]=bannerLinks.icon&populate[product][populate][2]=overview&populate[product][populate][3]=quickstart_guide&populate[product][populate][4]=release_note&populate[product][populate][5]=api_data_list_page&populate[product][populate][6]=api_data_list_page.apiData.*&populate[product][populate][7]=api_data_list_page.apiData.apiRestDetail.slug&populate[product][populate][8]=api_data_list_page.apiData.apiRestDetail.specUrls&populate[product][populate][9]=api_data_list_page.apiData.apiSoapDetail.*&populate[product][populate][10]=guide_list_page&populate[product][populate][11]=tutorial_list_page&populate[seo][populate]=*,metaImage,metaSocial.image&pagination[pageSize]=1000&pagination[page]=1'
+    ),
+    // Guide list pages
+    getResponseFromStrapi(


You should use the typed function fetchFromStrapi

marcobottaro · 2025-10-07T14:23:20Z

packages/gitbook-docs/src/scripts/syncAllMetadata.ts

+    guides: guidesResult.data,
+    solutions: solutionsResult.data,
+    releaseNotes: releaseNotesResult.data,
+    guidesResponse: guidesResult.responseJson,
+    solutionsResponse: solutionsResult.responseJson,
+    releaseNotesResponse: releaseNotesResult.responseJson,
+    guideListPagesResponse,
+    solutionListPageResponse,


Is it necessary to distinguish between data, responseJson and full Response?

marcobottaro · 2025-10-07T14:25:37Z

packages/gitbook-docs/src/scripts/syncAllMetadata.ts

+async function cachedListS3Files(prefix: string): Promise<string[]> {
+  if (s3ListCache.has(prefix)) {
+    return s3ListCache.get(prefix)!;
+  }
+  const files = await listS3Files(prefix, S3_BUCKET_NAME!, getS3Client());
+  s3ListCache.set(prefix, files);
+  return files;
+}
+
+async function cachedDownloadS3File(filePath: string): Promise<string> {
+  if (s3FileCache.has(filePath)) {
+    return s3FileCache.get(filePath)!;
+  }
+  const content = await downloadS3File(
+    filePath,
+    S3_BUCKET_NAME!,
+    getS3Client()
+  );
+  s3FileCache.set(filePath, content);
+  return content;
+}


You should read files from file system

…mprove directory checks, and streamline file reading

github-actions · 2025-10-08T21:29:06Z

Jira Pull Request Link

This Pull Request refers to the following Jira issue DEV-2583

github-actions · 2025-10-08T21:29:06Z

Branch is not up to date with base branch

@tommaso1 it seems this Pull Request is not updated with base branch.
Please proceed with a merge or rebase to solve this.

github-actions · 2025-10-08T21:29:11Z

This PR exceeds the recommended size of 800 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

marcobottaro · 2025-10-10T10:19:07Z

packages/gitbook-docs/src/scripts/syncAllMetadata.ts

+  ] = await Promise.all([
+    // Guides with full populate
+    fetchFromStrapi<StrapiGuide>(
+      'api/guides?populate[0]=product&populate[1]=versions&populate[2]=image&populate[3]=mobileImage&populate[4]=bannerLinks&populate[5]=bannerLinks.icon&populate[6]=listItems&populate[7]=seo&populate[8]=seo.metaImage&populate[9]=seo.metaSocial.image&pagination[pageSize]=1000&pagination[page]=1'


Here we should use Query String's stringify function like in apps/nextjs-website/src/lib/strapi/fetches/fetchGuides.ts.
By doing so, we could manage this query params by declaring a const like in this example:

populate: { image: { populate: '*' }, mobileImage: { populate: '*' }, listItems: { populate: '*' }, versions: { populate: '*' }, bannerLinks: { populate: ['icon'] }, seo: { populate: 'metaSocial.image' }, product: { ...productRelationsPopulate, }, }, };``` and then converting it to a string: `qs.stringify({ ...guidesPopulate, });`

marcobottaro · 2025-10-10T10:21:20Z

packages/gitbook-docs/src/scripts/syncAllMetadata.ts

+  ] = await Promise.all([
+    // Guides with full populate
+    fetchFromStrapi<StrapiGuide>(
+      'api/guides?populate[0]=product&populate[1]=versions&populate[2]=image&populate[3]=mobileImage&populate[4]=bannerLinks&populate[5]=bannerLinks.icon&populate[6]=listItems&populate[7]=seo&populate[8]=seo.metaImage&populate[9]=seo.metaSocial.image&pagination[pageSize]=1000&pagination[page]=1'


You should add &populate[product][populate][10]=use_case_list_page

This is the current and updated query:

api/guides/?populate[image][populate]=*&populate[mobileImage][populate]=*&populate[listItems][populate]=*&populate[versions][populate]=*&populate[bannerLinks][populate][0]=icon&populate[seo][populate]=metaSocial.image&populate[product][populate][0]=logo&populate[product][populate][1]=bannerLinks.icon&populate[product][populate][2]=overview&populate[product][populate][3]=quickstart_guide&populate[product][populate][4]=release_note&populate[product][populate][5]=api_data_list_page&populate[product][populate][6]=api_data_list_page.apiData.*&populate[product][populate][7]=api_data_list_page.apiData.apiRestDetail.*&populate[product][populate][8]=guide_list_page&populate[product][populate][9]=tutorial_list_page&populate[product][populate][10]=use_case_list_page

Sebastiano-Bertolin · 2025-10-10T10:52:19Z

packages/gitbook-docs/src/scripts/syncAllMetadata.ts

+    ),
+    // Guide list pages
+    fetchFromStrapi<StrapiGuideListPageResponse>(
+      'api/guide-list-pages/?populate%5Bproduct%5D%5Bpopulate%5D%5B0%5D=logo&populate%5Bproduct%5D%5Bpopulate%5D%5B1%5D=bannerLinks.icon&populate%5Bproduct%5D%5Bpopulate%5D%5B2%5D=overview&populate%5Bproduct%5D%5Bpopulate%5D%5B3%5D=quickstart_guide&populate%5Bproduct%5D%5Bpopulate%5D%5B4%5D=release_note&populate%5Bproduct%5D%5Bpopulate%5D%5B5%5D=api_data_list_page&populate%5Bproduct%5D%5Bpopulate%5D%5B6%5D=api_data_list_page.apiData.%2A&populate%5Bproduct%5D%5Bpopulate%5D%5B7%5D=api_data_list_page.apiData.apiRestDetail.%2A&populate%5Bproduct%5D%5Bpopulate%5D%5B8%5D=guide_list_page&populate%5Bproduct%5D%5Bpopulate%5D%5B9%5D=tutorial_list_page&populate%5BguidesByCategory%5D%5Bpopulate%5D%5B0%5D=guides.mobileImage&populate%5BguidesByCategory%5D%5Bpopulate%5D%5B1%5D=guides.image&populate%5BguidesByCategory%5D%5Bpopulate%5D%5B2%5D=guides.listItems&populate%5BbannerLinks%5D%5Bpopulate%5D%5B0%5D=icon&populate%5Bseo%5D%5Bpopulate%5D=%2A%2CmetaImage%2CmetaSocial.image'


Same for StrapiGuideListPage:

api/guide-list-pages/?populate[product][populate][0]=logo&populate[product][populate][1]=bannerLinks.icon&populate[product][populate][2]=overview&populate[product][populate][3]=quickstart_guide&populate[product][populate][4]=release_note&populate[product][populate][5]=api_data_list_page&populate[product][populate][6]=api_data_list_page.apiData.*&populate[product][populate][7]=api_data_list_page.apiData.apiRestDetail.*&populate[product][populate][8]=guide_list_page&populate[product][populate][9]=tutorial_list_page&populate[product][populate][10]=use_case_list_page&populate[guidesByCategory][populate][0]=guides.mobileImage&populate[guidesByCategory][populate][1]=guides.image&populate[guidesByCategory][populate][2]=guides.listItems&populate[bannerLinks][populate][0]=icon&populate[seo][populate]=*,metaImage,metaSocial.image

Sebastiano-Bertolin · 2025-10-10T10:54:00Z

packages/gitbook-docs/src/scripts/syncAllMetadata.ts

+    ),
+    // Release notes with full populate
+    fetchFromStrapi<StrapiReleaseNote>(
+      'api/release-notes/?populate[bannerLinks][populate][0]=icon&populate[product][populate][0]=logo&populate[product][populate][1]=bannerLinks.icon&populate[product][populate][2]=overview&populate[product][populate][3]=quickstart_guide&populate[product][populate][4]=release_note&populate[product][populate][5]=api_data_list_page&populate[product][populate][6]=api_data_list_page.apiData.*&populate[product][populate][7]=api_data_list_page.apiData.apiRestDetail.slug&populate[product][populate][8]=api_data_list_page.apiData.apiRestDetail.specUrls&populate[product][populate][9]=api_data_list_page.apiData.apiSoapDetail.*&populate[product][populate][10]=guide_list_page&populate[product][populate][11]=tutorial_list_page&populate[seo][populate]=*,metaImage,metaSocial.image&pagination[pageSize]=1000&pagination[page]=1'


This is the updated query with use_case_list_page:

api/release-notes/?populate[bannerLinks][populate][0]=icon&populate[product][populate][0]=logo&populate[product][populate][1]=bannerLinks.icon&populate[product][populate][2]=overview&populate[product][populate][3]=quickstart_guide&populate[product][populate][4]=release_note&populate[product][populate][5]=api_data_list_page&populate[product][populate][6]=api_data_list_page.apiData.*&populate[product][populate][7]=api_data_list_page.apiData.apiRestDetail.slug&populate[product][populate][8]=api_data_list_page.apiData.apiRestDetail.specUrls&populate[product][populate][9]=api_data_list_page.apiData.apiSoapDetail.*&populate[product][populate][10]=guide_list_page&populate[product][populate][11]=tutorial_list_page&populate[product][populate][12]=use_case_list_page&populate[seo][populate]=*,metaImage,metaSocial.image&pagination[pageSize]=1000&pagination[page]=1

github-user-name added 2 commits September 30, 2025 14:58

Add optimized GitBook sync workflow and metadata synchronization script

1b8977e

autolaunch

e6a384e

tommaso1 requested review from a team, MarcoPonchia and marcobottaro as code owners September 30, 2025 13:11

github-actions bot added changeset packages workflows size/large labels Sep 30, 2025

remove trigger

5f02245

marcobottaro requested changes Oct 7, 2025

View reviewed changes

github-user-name added 2 commits October 8, 2025 21:36

Add Strapi response interfaces for guide and solution list pages

3ccea38

Refactor metadata synchronization script: remove unused S3 caching, i…

c20a346

…mprove directory checks, and streamline file reading

marcobottaro requested changes Oct 10, 2025

View reviewed changes

Sebastiano-Bertolin requested changes Oct 10, 2025

View reviewed changes

	let s3Client: ReturnType<typeof makeS3Client> \| undefined;
	let s3Client: S3Client \| undefined;


		let s3Client: ReturnType<typeof makeS3Client> \| undefined;

		function getS3Client(): ReturnType<typeof makeS3Client> {

[DEV-2583] Add optimized GitBook sync workflow and metadata synchronization script #1711

Are you sure you want to change the base?

[DEV-2583] Add optimized GitBook sync workflow and metadata synchronization script #1711

Uh oh!

Conversation

tommaso1 commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

List of Changes

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

Uh oh!

changeset-bot bot commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 8, 2025 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Jira Pull Request Link

Uh oh!

github-actions bot commented Oct 8, 2025

Branch is not up to date with base branch

Uh oh!

github-actions bot commented Oct 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tommaso1 commented Sep 30, 2025 •

edited

Loading

changeset-bot bot commented Sep 30, 2025 •

edited

Loading

github-actions bot commented Oct 8, 2025 •

edited by atlassian bot

Loading