draft: [CAI-204] Chatbot/presidio in docker for local development #1201

batdevis · 2024-10-14T15:59:04Z

List of Changes

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Chore (nothing changes by a user perspective)
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

My change requires a change to the documentation.
I have updated the documentation accordingly.

…mpose

…query-by-date

Co-authored-by: marcobottaro <[email protected]>

…eveloper-portal into chatbot/docker-compose-complete

changeset-bot · 2024-10-14T15:59:08Z

⚠️ No Changeset found

Latest commit: b3c8a33

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

…mpose

Co-authored-by: marcobottaro <[email protected]>

mdciri · 2024-10-22T15:19:02Z

apps/chatbot/docker/app.local.Dockerfile


-CMD ["fastapi", "dev", "src/app/main.py", "--port", "8080"]
+RUN python -m spacy download it_core_news_md
+RUN python -m spacy download en_core_web_md


here are missing to download: es_core_news_md, de_core_news_md, fr_core_news_md, el_core_news_md, hr_core_news_md, and sl_core_news_md.

mdciri · 2024-10-22T15:26:34Z

apps/chatbot/pyproject.toml

+chromedriver-py = "^129.0.6668.91"
 llama-index-postprocessor-presidio = "^0.2.0"
+presidio-anonymizer = "^2.2.355"
+presidio-analyzer = "^2.2.355"


remove presidio-anonymizer and presidio-analyzer and install llama-index-postprocessor-presidio as in the pyproject.toml in main. Check this out.

In the end, why chromedriver-py?

mdciri · 2024-10-22T15:29:35Z

apps/chatbot/src/app/main.py

-  profile_name = locals().get('profile_name', None),
-  region_name=locals().get('region_name', None)
+  region_name=AWS_DEFAULT_REGION
 )


in boto3 session are missing the aws_access_key_id and aws_secret_access_key

mdciri · 2024-10-22T15:31:12Z

apps/chatbot/src/app/main.py

  queriedAt: str | None = None

-if (os.getenv('environment', 'dev') == 'local'):
-  profile_name='dummy'


I saw this dummy in many files. isn't better to but this profile_name in config/params.yaml or in a enviroment variable?

mdciri · 2024-10-22T15:32:29Z

apps/chatbot/src/app/main.py

+    'dynamodb',
+    endpoint_url=os.getenv('CHB_DYNAMODB_URL', 'http://localhost:8000'),
+    region_name=AWS_DEFAULT_REGION
+  )


aws_access_key_id and aws_secret_access_key are missing

mdciri · 2024-10-22T15:32:34Z

apps/chatbot/src/app/main.py

+    'dynamodb',
+    region_name=AWS_DEFAULT_REGION
+  )



aws_access_key_id and aws_secret_access_key are missing

mdciri · 2024-10-22T15:33:27Z

apps/chatbot/src/modules/utils.py

+
    ssm = boto3.client(
        "ssm",
        aws_access_key_id=AWS_ACCESS_KEY_ID,


aws_access_key_id and aws_secret_access_key are missing, but this is already handled in main

mdciri · 2024-10-22T15:35:14Z

apps/chatbot/src/modules/vector_database.py

-        if file in dynamic_htmls:
+# FIX: resolve webdriver.Chrome "self.assert_process_still_running" error in docker
+#        if file in dynamic_htmls:
+        if 6 == 9:


github-actions · 2024-10-23T13:46:53Z

Jira Pull Request Link

This Pull Request refers to the following Jira issue CAI-204

batdevis · 2024-10-30T09:27:52Z

now it works with Presidio

batdevis and others added 26 commits October 8, 2024 15:38

feat(chatbot): session GSI

ac9b39a

feat(chatbot): docker compose

36c89af

fix(chatbot): dynamodb and redis for local development with docker co…

a2c16c7

…mpose

Merge branch 'chatbot/docker-compose-complete' into chatbot/sessions-…

c633df6

…query-by-date

chore(chatbot):remove duplicate imports

08a7fec

chore(chatbot): linting

7fe9101

fix(chatbot):create index in docker

9b2eb61

chore(chatbot): llamaindex index id

b251837

fix(chatbot): create vector index with all docs

4ecd4b3

Merge branch 'main' into chatbot/docker-compose-complete

a84581c

chore(chatbot): terraform lint

ea7d3db

fix(chatbot): terraform syntax

28695e3

chore(chatbot): remove dynamodb options

238edfd

chore(chatbot): from global to local secondary index

5f63560

Merge branch 'main' into chatbot/docker-compose-complete

859c298

chore: merge main

f43a771

chore: remove old var

d96a9f9

chore: merge main

9526c17

Update apps/chatbot/docker/compose.yaml

a5177df

Co-authored-by: marcobottaro <[email protected]>

chore: remove logs

4daccf8

Merge branch 'chatbot/docker-compose-complete' of github.com:pagopa/d…

f5bdfd3

…eveloper-portal into chatbot/docker-compose-complete

fix(chatbot): compose vars

c123b5c

Merge branch 'main' into chatbot/docker-compose-complete

aa59ca5

fix(chatbot): presidio in docker

d25c297

chore(chatbot): spacy dockerfile

ef0f916

fix(chatbot): download presidio models in docker

57d712c

batdevis requested a review from a team as a code owner October 14, 2024 15:59

batdevis temporarily deployed to prod October 14, 2024 15:59 — with GitHub Actions Inactive

batdevis temporarily deployed to dev October 14, 2024 15:59 — with GitHub Actions Inactive

batdevis and others added 18 commits October 16, 2024 16:36

feat(chatbot): session GSI

e5aab9a

feat(chatbot): docker compose

5c449d3

fix(chatbot): dynamodb and redis for local development with docker co…

f7d2709

…mpose

chore(chatbot): linting

3ab9e7c

fix(chatbot):create index in docker

e1ae526

fix(chatbot): create vector index with all docs

fc32f78

chore(chatbot): terraform lint

dde1f40

fix(chatbot): terraform syntax

db11504

chore(chatbot): remove dynamodb options

acf2a01

chore(chatbot): from global to local secondary index

a78a213

chore: remove old var

5d66ef5

chore: remove logs

5f7e997

Update apps/chatbot/docker/compose.yaml

2d6b242

Co-authored-by: marcobottaro <[email protected]>

fix(chatbot): compose vars

7235e4f

fix(chatbot): presidio in docker

a5bd612

chore(chatbot): spacy dockerfile

040eabc

fix(chatbot): download presidio models in docker

66e2a69

fix: make docker-compose work with presidio

d97640a

christian-calabrese force-pushed the chatbot/presidio-in-docker branch from 57d712c to d97640a Compare October 16, 2024 15:27

christian-calabrese temporarily deployed to prod October 16, 2024 15:27 — with GitHub Actions Inactive

christian-calabrese temporarily deployed to dev October 16, 2024 15:27 — with GitHub Actions Inactive

batdevis changed the title ~~draft: [CAI-204] Chatbot/presidio in docker for local development~~ [CAI-204] Chatbot/presidio in docker for local development Oct 21, 2024

batdevis changed the title ~~[CAI-204] Chatbot/presidio in docker for local development~~ draft: [CAI-204] Chatbot/presidio in docker for local development Oct 21, 2024

mdciri requested changes Oct 22, 2024

View reviewed changes

batdevis added 2 commits October 23, 2024 15:41

chore: merge main

7cc252a

chore: merge main

b3c8a33

github-actions bot removed the infra label Oct 23, 2024

batdevis closed this Oct 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

draft: [CAI-204] Chatbot/presidio in docker for local development #1201

draft: [CAI-204] Chatbot/presidio in docker for local development #1201

Uh oh!

batdevis commented Oct 14, 2024

Uh oh!

changeset-bot bot commented Oct 14, 2024 •

edited

Loading

Uh oh!

mdciri Oct 22, 2024

Uh oh!

mdciri Oct 22, 2024

Uh oh!

mdciri Oct 22, 2024

Uh oh!

mdciri Oct 22, 2024

Uh oh!

mdciri Oct 22, 2024

Uh oh!

mdciri Oct 22, 2024

Uh oh!

mdciri Oct 22, 2024

Uh oh!

mdciri Oct 22, 2024

Uh oh!

github-actions bot commented Oct 23, 2024 •

edited by atlassian bot

Loading

Uh oh!

batdevis commented Oct 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

draft: [CAI-204] Chatbot/presidio in docker for local development #1201

draft: [CAI-204] Chatbot/presidio in docker for local development #1201

Uh oh!

Conversation

batdevis commented Oct 14, 2024

List of Changes

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

Checklist:

Uh oh!

changeset-bot bot commented Oct 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 23, 2024 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Jira Pull Request Link

Uh oh!

batdevis commented Oct 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

changeset-bot bot commented Oct 14, 2024 •

edited

Loading

github-actions bot commented Oct 23, 2024 •

edited by atlassian bot

Loading