[CAI-445] - API documents in the index #1466

mdciri · 2025-06-09T08:18:28Z

List of Changes

Update the modules to read also the API from the openapi files

Motivation and Context

in this way the chatbot can answer properly the queries about the APIs

How Has This Been Tested?

locally

Screenshots (if appropriate):

Types of changes

Chore (nothing changes by a user perspective)
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

My change requires a change to the documentation.
I have updated the documentation accordingly.

changeset-bot · 2025-06-09T08:18:31Z

🦋 Changeset detected

Latest commit: 8df95bb

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
chatbot	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

apps/chatbot/src/modules/chatbot.py

apps/chatbot/src/modules/documents.py

apps/chatbot/src/modules/chatbot.py

apps/chatbot/src/modules/create_vector_index.py

github-actions · 2025-06-12T10:53:30Z

Branch is not up to date with base branch

@mdciri it seems this Pull Request is not updated with base branch.
Please proceed with a merge or rebase to solve this.

github-actions · 2025-06-12T14:41:46Z

Jira Pull Request Link

This Pull Request refers to the following Jira issue CAI-445

github-actions · 2025-06-12T14:41:46Z

Jira Pull Request Link

This Pull Request refers to the following Jira issue CAI-445

* Update modules to read API docs and put them into the index * Update main prompt * Update modules and set sentence splitter as node parser * Update notebook * Update params for the sentence splitter node parser * Add changeset * Update documents.py to use strapi api key * Update chatbot.py * Update vector_database.py * Update engine.py * Update engine.py * Update models.py * Update creation index script * fixed typo * Update strapi env var from SSM * Update chatbot.py * Update documents.py * Update documents.py --------- Co-authored-by: marcobottaro <[email protected]>

* Use standalone * Set reactStrictMode to false * fix: use env secret * fix: parse secrets temporarily * fix: linting * make dynamic pages: webinars, guide and soltuion * use env instead of secrets to building locally * refactor guide using S3 * Fix env var management * remove sitemap * Add verbose build info * add env var logs * add log * move log * Update package json * update config * Fix config * [DEV-2132] Poc standalone improve s3 fetch guide (#1354) * Fix CICD lambda permissions (#1353) * fix: lambda permissions arn * docs: added changeset * improve fetch guide from S3 * convert all gitbook page using s3 * Fix images and solutions page * Fix after review * Add changeset --------- Co-authored-by: Christian Calabrese <[email protected]> * [DEV-2184] Poc standalone fix guide version (#1365) * Fix guide version and page title * Fix release note pages * add changeset * Add debug logs * Update logs * Use secret for bucket * update yaml * Fix fetch from strapi using secrets * Fix fetch from strapi using secrets * remove debug log * Fix build locallu * [DEV-2129] Sitemap for standalone version (#1368) * Add sitemap generation scripts for dynamic content from S3 and Strapi * Update apps/nextjs-website/scripts/createSitemapJson.ts Co-authored-by: Marco Ponchia <[email protected]> * Update apps/nextjs-website/scripts/createSitemapJson.ts Co-authored-by: Marco Ponchia <[email protected]> * Update apps/nextjs-website/scripts/createSitemapJson.ts Co-authored-by: Marco Ponchia <[email protected]> * Update apps/nextjs-website/scripts/createSitemapJson.ts Co-authored-by: Marco Ponchia <[email protected]> * Update apps/nextjs-website/scripts/createSitemapJson.ts Co-authored-by: Marco Ponchia <[email protected]> * pr comments * Refactor sitemap generation to use new environment variable for guide metadata path and add script for generating guide metadata from S3 and Strapi. * sitemap --------- Co-authored-by: t <[email protected]> Co-authored-by: Marco Ponchia <[email protected]> Co-authored-by: marcobottaro <[email protected]> * Fix sitemap * [DEV-2196] Update generate metadata functions (#1392) * add metadata generator helpers * refactor guide script * add new generate scripts and update package * update sitemap * Fix after review * Add changeset * Fix after review * [DEV-2226] Fix doc s3 parser (#1407) * Fix parseS3Doc helper * add changeset * Update shaggy-lizards-lick.md Update changeset --------- Co-authored-by: marcobottaro <[email protected]> * [DEV-2131] fix dompurify warning (#1418) * Fix dompurify warning * Add changeset --------- Co-authored-by: marcobottaro <[email protected]> * [DEV-2243] parse single guide page (#1411) * add metadata generator helpers * refactor guide script * add new generate scripts and update package * update sitemap * Fix after review * Add changeset * Add parseS3DocPage helper * Update sitemap and s3Metadata helpers * Add getGuidePage and getGuidePageProps * Update page WIP * Fix menu path and update guide version logic * Update JsonMetatadata fields * Update parseS3GuidePage * Fix api.ts * Refactor cmsApi * Update guide metadata page WIP * Add cache for metadata fetches * Fix after review * Add changeset * Fix parse error * Add package-lock.json updates --------- Co-authored-by: Marco Bottaro <[email protected]> * Add sst configuration * replaced aplifiy with open next new module * refactory condition with for static parts * module outputs and first draft stendalone deploy action * update github action permission and bug fixing * update build commands * [DEV-2311] add basic isr to next (#1450) * Add basic ISR to pages * Add ISR to guide pages and build statically first pagge for guide from CMS * Add changeset * Update apps/nextjs-website/src/app/[productSlug]/[...releaseNoteSubPathSlugs]/page.tsx Co-authored-by: marcobottaro <[email protected]> --------- Co-authored-by: marcobottaro <[email protected]> * Improve debug logs * update package json file * Add session token to client s3 credentials * add force dynamic * update deploy standalone action with permissions * deploy standalone assign aws env variables * deploy standalone setup node npm cache * Remove force dynamic * remove guide entrypoint static generation * update deploy stanalone action * deploy standalone copy assets, cache and zip all functions * update deploy lambda policy, s3 bucket to deploy code, deploy opennext lambda * update deploy standalone action * update action deploy standalone: deploy all functions * git action deploy standalone, remvoved the package part * update iam policy to deploy opennext * deploy standalone create new version and update alias * fix deploy standalone action cache step * update invoke command: it requires an outfile param * website cdn new s3 origin for standalone content * update bucket content policy and cloud front aliases * Fix metadata cache * update lambda server env variables * [DEV-2323] Fetch data using CDN (#1462) * Add env var * Update fetching data helper to fetch from cdn instead of bucket * Refactor parser of S3 doc to use new env var end helpers * Add changeset * Remove console log * terrform lock file update * fix linting error * Enable cacheInterception * open next lambda new env var: NEXT_PUBLIC_FEEDBACK_FORM_ENABLED * new strapi secret in parameter store * fix terraform template and secret parameter * [DEV-2324] Fix absolute url path transform (#1472) * WIP Add test to fix * Fix absolute url path transform --------- Co-authored-by: Marco Ponchia <[email protected]> * Fix linting * update caching policy for static content cdn * set cors configuration for static-contents * update cloufront cors headers with full domains * alarm for opennext lambda errors * config opennext instrumentation * config opennext instrumentation * instrumentation configs * fix linting * rollback confgurations to instrument next.js with telemetry * rollback package.json nextjs website * [DEV-2350] Add logs in guide pages (#1476) * Add logs to guide page fetch * add changeset * update iam policy to invalidate cloudfront opennext cache * update action deploy standalow: clean up configurations * [CAI-437] - Fix user feedback bug (#1459) * Fix user feedback bug * Add changeset * Add changeset * Remove double changeset --------- Co-authored-by: marcobottaro <[email protected]> * [CAI-445] - API documents in the index (#1466) * Update modules to read API docs and put them into the index * Update main prompt * Update modules and set sentence splitter as node parser * Update notebook * Update params for the sentence splitter node parser * Add changeset * Update documents.py to use strapi api key * Update chatbot.py * Update vector_database.py * Update engine.py * Update engine.py * Update models.py * Update creation index script * fixed typo * Update strapi env var from SSM * Update chatbot.py * Update documents.py * Update documents.py --------- Co-authored-by: marcobottaro <[email protected]> * [DEV-2344] Fix call order in fetch local docs (#1474) * Fix fetch-local-docs * Add changeset --------- Co-authored-by: marcobottaro <[email protected]> * rebase poc stanalone * Run standalone build in self hosted runner * [DEV-2347] Convert to static pages coming from gitbook (#1473) * add to static params page from gitbook * add changeset * Fix gitbook parsing for solutions and release notes * Fix release note pages * remove console log * fix generate metadata skip summary * Add changeset * Fix after review * [DEV-2342] Add generic nextjs error (#1486) * Add generic error * add changeset * Fix changeset --------- Co-authored-by: Marco Bottaro <[email protected]> Co-authored-by: christian-calabrese <[email protected]> Co-authored-by: tommaso1 <[email protected]> Co-authored-by: t <[email protected]> Co-authored-by: marcobottaro <[email protected]> Co-authored-by: uolter <[email protected]> Co-authored-by: Marcello Bertoli <[email protected]> Co-authored-by: MarBert <[email protected]> Co-authored-by: Marco Domenico Cirillo <[email protected]>

* Update modules to read API docs and put them into the index * Update main prompt * Update modules and set sentence splitter as node parser * Update notebook * Update params for the sentence splitter node parser * Add changeset * Update documents.py to use strapi api key * Update chatbot.py * Update vector_database.py * Update engine.py * Update engine.py * Update models.py * Update creation index script * fixed typo * Update strapi env var from SSM * Update chatbot.py * Update documents.py * Update documents.py --------- Co-authored-by: marcobottaro <[email protected]>

mdciri added 2 commits June 9, 2025 10:12

Update modules to read API docs and put them into the index

0ff7c06

Update main prompt

b687a3e

mdciri requested review from a team, MarcoPonchia, marcobottaro and tommaso1 as code owners June 9, 2025 08:18

mdciri self-assigned this Jun 9, 2025

mdciri requested a review from batdevis June 9, 2025 08:18

mdciri and others added 5 commits June 9, 2025 10:18

Merge branch 'main' into CAI-445-api-documentation

6c164cd

Update modules and set sentence splitter as node parser

6ec0095

Update notebook

2153dbc

Update params for the sentence splitter node parser

834c536

Add changeset

232a4a6

github-actions bot added the changeset label Jun 9, 2025

Update documents.py to use strapi api key

84cc32c

batdevis reviewed Jun 10, 2025

View reviewed changes

apps/chatbot/src/modules/chatbot.py Show resolved Hide resolved

mdciri and others added 6 commits June 10, 2025 11:44

Update chatbot.py

7790710

Update vector_database.py

7f786b4

Update engine.py

230f5a9

Update engine.py

6e577a9

Update models.py

f4283ac

Merge branch 'main' into CAI-445-api-documentation

4e9a98d

mdciri requested a review from batdevis June 10, 2025 10:05

mdciri and others added 3 commits June 10, 2025 14:45

Update creation index script

a3c6ec1

fixed typo

16d8ef7

Merge branch 'main' into CAI-445-api-documentation

544ca39

marcobottaro mentioned this pull request Jun 10, 2025

[CAI-451] Chatbot Logs #1468

Merged

6 tasks

Update strapi env var from SSM

0798165

marcobottaro reviewed Jun 10, 2025

View reviewed changes

apps/chatbot/src/modules/documents.py Show resolved Hide resolved

mdciri added 2 commits June 11, 2025 17:29

Update chatbot.py

6aa9559

Update documents.py

3a697e9

batdevis approved these changes Jun 11, 2025

View reviewed changes

apps/chatbot/src/modules/chatbot.py Show resolved Hide resolved

apps/chatbot/src/modules/create_vector_index.py Outdated Show resolved Hide resolved

Update documents.py

8f6f148

marcobottaro added 2 commits June 12, 2025 16:41

Merge branch 'main' into CAI-445-api-documentation

66e3f9f

Merge branch 'main' into CAI-445-api-documentation

8df95bb

marcobottaro approved these changes Jun 12, 2025

View reviewed changes

marcobottaro merged commit e0705a8 into main Jun 12, 2025
12 checks passed

marcobottaro deleted the CAI-445-api-documentation branch June 12, 2025 14:43

marcobottaro temporarily deployed to dev June 12, 2025 14:43 — with GitHub Actions Inactive

marcobottaro had a problem deploying to dev June 12, 2025 14:43 — with GitHub Actions Failure

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[CAI-445] - API documents in the index #1466

[CAI-445] - API documents in the index #1466

Uh oh!

mdciri commented Jun 9, 2025

Uh oh!

changeset-bot bot commented Jun 9, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jun 12, 2025

Uh oh!

github-actions bot commented Jun 12, 2025 •

edited by atlassian bot

Loading

Uh oh!

github-actions bot commented Jun 12, 2025 •

edited by atlassian bot

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[CAI-445] - API documents in the index #1466

[CAI-445] - API documents in the index #1466

Uh oh!

Conversation

mdciri commented Jun 9, 2025

List of Changes

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

Uh oh!

changeset-bot bot commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jun 12, 2025

Branch is not up to date with base branch

Uh oh!

github-actions bot commented Jun 12, 2025 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Jira Pull Request Link

Uh oh!

github-actions bot commented Jun 12, 2025 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Jira Pull Request Link

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

changeset-bot bot commented Jun 9, 2025 •

edited

Loading

github-actions bot commented Jun 12, 2025 •

edited by atlassian bot

Loading

github-actions bot commented Jun 12, 2025 •

edited by atlassian bot

Loading