[CAI-741][CAI-762] Chatbot-index creates new indexes on Redis by mdciri · Pull Request #2008 · pagopa/developer-portal

mdciri · 2026-02-09T17:02:41Z

List of Changes

This pull request introduces several enhancements and new features to the chatbot indexer, with a focus on supporting structured data. The most important changes include adding support for structured document indexing, refactoring the document ingestion and index creation workflow to be more flexible.

Added support for indexing structured documents by introducing a new get_structured_docs() function and updating the main document ingestion logic to accept flags for including static, dynamic, API, and structured data.
Refactored the index creation process to use command-line arguments for specifying which types of data to include, and updated the GitHub Actions workflow and composite action to pass these options through inputs.
Enhanced the Redis index build process to allow for selective cleaning of index data, and made the schema and index ID configurable via environment variables and parameters.

Motivation and Context

This refactor should allow the creation of new vector indexes on Redis

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Chore (nothing changes by a user perspective)
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

My change requires a change to the documentation.
I have updated the documentation accordingly.

* Add package.json to parser app * Add changeset, update package-lock * Update apps/parser/package.json Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add package.json to parser app * Add changeset, update package-lock * Update apps/parser/package.json Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add puppeteer dependency in parser app * Add changeset * Update .changeset/three-cups-change.md Co-authored-by: marcobottaro <39835990+marcobottaro@users.noreply.github.com> * Update apps/parser/package.json Co-authored-by: marcobottaro <39835990+marcobottaro@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: marcobottaro <39835990+marcobottaro@users.noreply.github.com>

…ndex

changeset-bot · 2026-02-09T17:02:46Z

🦋 Changeset detected

Latest commit: 6c65f57

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
chatbot-index	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Copilot

Pull request overview

Refactors the chatbot indexer to support configurable, selective document ingestion (including structured docs) and Redis index creation/cleanup, while updating GitHub Actions inputs to drive the new CLI flags.

Changes:

Added structured document ingestion and flag-based document selection for indexing.
Refactored Redis index schema/creation/loading to support configurable index_id and targeted cleanup.
Updated workflow/action wiring and added Terraform skill documentation files.

Reviewed changes

Copilot reviewed 14 out of 15 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
apps/parser/package.json	Adds a new `parser` app package manifest with Puppeteer dependency.
apps/chatbot-index/src/modules/vector_index.py	Introduces schema factory + parameterized index build/load/cleanup behavior.
apps/chatbot-index/src/modules/settings.py	Switches `index_id` configuration to environment variable.
apps/chatbot-index/src/modules/documents.py	Adds structured docs ingestion and flags to select doc sources.
apps/chatbot-index/src/modules/create_vector_index.py	Adds CLI flags to choose which document types to include and whether to clean Redis.
apps/chatbot-index/src/modules/codec.py	Adds helper to safely parse JSON-encoded strings.
apps/chatbot-index/config/params.yaml	Removes `index_id` from params configuration.
.github/workflows/chatbot_create_index.yaml	Adds workflow input to choose indexing mode; passes inputs to composite action.
.github/actions/chatbot/action.yaml	Adds composite action inputs and passes corresponding CLI flags to indexer.
.github/skills/terraform-style-guide/SKILL.md	Adds Terraform style guide documentation.
.github/skills/terraform-refactor-module/SKILL.md	Adds Terraform module refactor skill documentation.
.changeset/three-cups-change.md	Changeset for parser Puppeteer addition.
.changeset/eager-colts-smile.md	Changeset for chatbot-index structured docs support.
.changeset/cuddly-pumas-cross.md	Changeset for creating parser app.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

apps/chatbot-index/src/modules/settings.py

apps/chatbot-index/src/modules/documents.py

apps/chatbot-index/src/modules/create_vector_index.py

.github/workflows/chatbot_create_index.yaml

.github/actions/chatbot/action.yaml

apps/chatbot-index/src/modules/codec.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 8 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

apps/chatbot-index/src/modules/vector_index.py

apps/chatbot-index/src/modules/create_vector_index.py

apps/chatbot-index/src/modules/documents.py

.github/workflows/chatbot_create_index.yaml

.github/skills/terraform-refactor-module/SKILL.md

.github/skills/terraform-style-guide/SKILL.md

.changeset/eager-colts-smile.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…veloper-portal into CAI-741-refactor-for-new-index

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

batdevis · 2026-02-10T14:21:17Z

apps/chatbot-index/src/modules/documents.py

why use both AWS client and AWS resource for S3?

I think it is possible to use only AWS resourse.

batdevis · 2026-02-10T15:02:20Z

apps/chatbot-index/src/modules/create_vector_index.py

+import argparse
+
 from src.modules.logger import get_logger
 from src.modules.vector_index import DiscoveryVectorIndex


DiscoveryVectorIndex is not present in src.modules.vector_index

I have an idea, you could rename the class : )

Copilot

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

apps/chatbot-index/src/modules/vector_index.py

apps/chatbot-index/src/modules/documents.py

apps/chatbot-index/src/modules/codec.py

apps/chatbot-index/src/modules/documents.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…veloper-portal into CAI-741-refactor-for-new-index

github-actions · 2026-02-11T16:22:28Z

Jira Pull Request Link

This Pull Request refers to the following Jira issue CAI-741

github-actions · 2026-02-11T16:22:30Z

Branch is not up to date with base branch

@batdevis it seems this Pull Request is not updated with base branch.
Please proceed with a merge or rebase to solve this.

github-actions · 2026-02-11T16:22:33Z

This PR exceeds the recommended size of 800 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

mdciri and others added 8 commits February 5, 2026 16:23

Refactor for multi index in redis

56cbc91

Copilot terraform skills

a0fe349

[CAI-747] add parser app (#1996)

24ad7a4

* Add package.json to parser app * Add changeset, update package-lock * Update apps/parser/package.json Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update scripts to accept structured documents to create a dedicated i…

e779b68

…ndex

Update github action accordingly to python script

15b8b95

Update chatbot create index workflow to accept input types

f096b08

Update documents to read structured data into llamaindex documents

062d3b7

mdciri requested review from batdevis and Copilot February 9, 2026 17:02

mdciri requested review from a team, MarBert, MarcoPonchia, marcobottaro and tommaso1 as code owners February 9, 2026 17:02

Copilot AI reviewed Feb 9, 2026

View reviewed changes

marcobottaro assigned mdciri Feb 10, 2026

mdciri and others added 6 commits February 10, 2026 14:32

Update apps/chatbot-index/src/modules/settings.py

af6dc9c

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update apps/chatbot-index/src/modules/documents.py

7128817

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update .github/actions/chatbot/action.yaml

17badb0

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Fix the input and output types

f50a388

Update structured llamaindex documents id

0df9e63

Add assertion for the inputs

338efd6

mdciri requested a review from Copilot February 10, 2026 13:44

Copilot AI reviewed Feb 10, 2026

View reviewed changes

mdciri and others added 4 commits February 10, 2026 14:48

Update .changeset/eager-colts-smile.md

d1a8d4d

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update apps/chatbot-index/src/modules/create_vector_index.py

57d348f

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update document to safe load jsons

4b315f0

Merge branch 'CAI-741-refactor-for-new-index' of github.com:pagopa/de…

01ed416

…veloper-portal into CAI-741-refactor-for-new-index

github-actions bot added the workflows label Feb 10, 2026

mdciri and others added 2 commits February 10, 2026 15:00

Update cleaning redis when creating index given an index id

74d9b35

Update .github/workflows/chatbot_create_index.yaml

8966f99

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

batdevis reviewed Feb 10, 2026

View reviewed changes

Sobstitute s3 client with s3 resource

6025cc3

mdciri changed the title ~~[CAI-741][CAI-762] Refactor of chatbot-index to create new indexes on Redis~~ [CAI-741][CAI-762] Chatbot-index creates new indexes on Redis Feb 10, 2026

batdevis reviewed Feb 10, 2026

View reviewed changes

mdciri requested review from batdevis and Copilot February 10, 2026 15:04

Copilot AI reviewed Feb 10, 2026

View reviewed changes

mdciri and others added 15 commits February 10, 2026 16:28

Update apps/chatbot-index/src/modules/vector_index.py

7a2232a

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update apps/chatbot-index/src/modules/documents.py

e2547c5

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update apps/chatbot-index/src/modules/codec.py

15187d3

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update documents json loads

bc02597

fix docker compose

149725b

Merge branch 'CAI-741-refactor-for-new-index' of github.com:pagopa/de…

ac5bd1f

…veloper-portal into CAI-741-refactor-for-new-index

Add to the github workflow and action the index-id to create as env var

4f70244

Merge branch 'CAI-741-refactor-for-new-index' of github.com:pagopa/de…

c3f2e8b

…veloper-portal into CAI-741-refactor-for-new-index

Update class name to be more generic

cda13ba

Update class name to be more generic

779967b

Merge branch 'main' into CAI-741-refactor-for-new-index

14e0a75

Update workflow inputs

873f81e

Update create-index service in compose.yaml

34dbc95

Update dockerfile and scripts

f3271e0

Update codec return

5b91eff

github-actions bot added the size/large label Feb 11, 2026

fix s3 init on create index

6c65f57

Conversation

mdciri commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

List of Changes

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

Uh oh!

changeset-bot bot commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

batdevis Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

batdevis Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

batdevis Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Feb 11, 2026 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Jira Pull Request Link

Uh oh!

github-actions bot commented Feb 11, 2026

Branch is not up to date with base branch

Uh oh!

github-actions bot commented Feb 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mdciri commented Feb 9, 2026 •

edited

Loading

changeset-bot bot commented Feb 9, 2026 •

edited

Loading

batdevis Feb 10, 2026 •

edited

Loading

github-actions bot commented Feb 11, 2026 •

edited by atlassian bot

Loading