Release/2025.07.1 [main] (#1917)

* feat: Enhances OneUp Health integration for document retrieval (M2-8883) (#1879)

* Enhances OneUp Health integration for document retrieval

Improves the OneUp Health integration by adding functionality
to download and store patient documents associated with EHR data.

This change introduces the ability to fetch documents referenced
in DocumentReference resources, store them in a zip file,
and upload the zip to the EHR storage.

It also introduces generic file uploading and listing capabilities
in the EHR storage, along with helper methods.

* Adds metadata to EHR answers

Adds metadata to the EHR answers table to store information about the uploaded zip files, including their names and sizes.

This allows for better tracking and management of the EHR data stored in the system.

* Adds assertion to task ingest user data

Adds an assertion to check that the result of the
`ingest_user_data` function is not None. This ensures that
the task completes successfully and returns a valid result.

* Update src/apps/integrations/oneup_health/service/oneup_health.py

Co-authored-by: Farmer Paul <paul.hh@metalab.com>

* Refactors EHR metadata handling

Updates EHR metadata to use a dedicated data model.

This change introduces `EHRFileMetadata` and `EHRFileTypeEnum`
to provide more structured and type-safe handling of EHR file
metadata, replacing the previous use of dictionaries. This improves
code clarity and maintainability.

* Adds provider name to EHR document filename

Ensures that the EHR document filename includes the
healthcare provider's name (or ID if name unavailable)
to improve identification and organization.

* Adds buckets for answers and operations

Extends MinIO bucket creation to include buckets for answers and operations, in addition to the existing media bucket.
This allows for a more organized storage structure within MinIO and improves separation of concerns.

* Adds a TODO item to optimize zip file creation.

Adds a TODO item to address potential memory issues
when creating zip files, especially when dealing with
large documents or numerous files.

* Refactors EHR file upload process

Streamlines the EHR file upload process by moving the base path
generation logic into the `EHRStorage.upload_file` method.
This change improves code maintainability and reduces redundancy.

---------

Co-authored-by: Farmer Paul <paul.hh@metalab.com>

* fix: Refactors ehr zip download to use storage path (M2-9443) (#1893)

Simplifies the ehr zip download process by directly using the storage path. This eliminates redundant path construction and improves code readability.

* chore: Added worker configuration to uvicorn startup (#1896)

* fix: Optimizes schedule event retrieval for performance (#1897)

* Optimizes event retrieval for schedule app

Improves performance by fetching events and associated
notifications/reminders in bulk using applet IDs.
This reduces database queries and speeds up schedule loading.

* Optimizes event data retrieval

Reduces the number of database queries required to fetch event
notifications and reminders by retrieving them in bulk.

This change improves performance when fetching schedule events,
especially when retrieving events for a large number of users.

* fix: Temp fix for workers (#1902)

* Fix: Enhances EHR exports to filter only allowed subjects answers (M2-9434) (#1894)

* Adjusts retry delay for task processing errors

Modifies the retry mechanism to use a linear delay
after the first failed attempt. This avoids excessive
wait times when processing tasks that encounter errors.

* Filters EHR answers by allowed subjects

Ensures that exported EHR answers are filtered based on the subjects the user is authorized to access.
If a list of allowed subjects is available, it's added as a filter when querying EHR answers.

* Refines subject ID filtering for EHR answers

Ensures that the target subject IDs for EHR answer exports are
filtered to only include IDs that are allowed for the user.
This prevents unauthorized access to data associated with subjects
outside the user's permitted scope.

* Adds reviewer role to EHR data endpoint test

Extends the EHR data endpoint test to include scenarios for
users with the reviewer role. This ensures that reviewers
can properly access and download EHR data relevant to their
responsibilities.

* Disabling APM for feature environments

* fix: Adds index to activity_items.activity_id (#1901)

Improves query performance by adding an index to the
activity_id column in the activity_items table.

Optimizes database lookups and enhances the efficiency
of related queries.

* [pip] Dependabot: Bump ddtrace from 2.21.8 to 3.9.4 (#1908)

Bumps [ddtrace](https://github.com/DataDog/dd-trace-py) from 2.21.8 to 3.9.4.
- [Release notes](https://github.com/DataDog/dd-trace-py/releases)
- [Changelog](https://github.com/DataDog/dd-trace-py/blob/main/CHANGELOG.md)
- [Commits](DataDog/dd-trace-py@v2.21.8...v3.9.4)

---
updated-dependencies:
- dependency-name: ddtrace
  dependency-version: 3.9.4
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [pip] Dependabot: Bump botocore from 1.38.28 to 1.38.46 (#1907)

---
updated-dependencies:
- dependency-name: botocore
  dependency-version: 1.38.46
  dependency-type: indirect
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [pip] Dependabot: Bump google-api-python-client from 2.170.0 to 2.174.0 (#1906)

---
updated-dependencies:
- dependency-name: google-api-python-client
  dependency-version: 2.174.0
  dependency-type: indirect
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Use the real exception message (#1904)

* chore: Removing workers (#1909)

* chore: Enable Database connection pooling (M2-9464) (#1898)

* chore: Enabled connection pooling

* fixed timeout and docs

* cq

* feat: Add logging around /answers 422 errors (M2-9475) (#1905)

* feat: Add logging around /answers 422 errors (M2-9475)

* Remove unused methods

* M2 9399 transfer ownership (#1888)

* Update errors.py

Adding the expected message error for the transfer ownvership test

* Update tests.py

Adding the new migrated test from TAF:
- Adding the fixture applet_one_bob_coordinator_reviewer
- Adding the tests test_manager_can_not_transfer_ownership
test_respondent_can_not_transfer_ownershipto
test_reviewer_can_not_transfer_ownership
test_editor_can_not_transfer_ownership
test_coordinator_can_not_transfer_ownership

* Update errors.py

deleting the error exception to use TransferOwnershipAccessDenied

* Update tests.py

deleting in all the test
        key = uuid.uuid4()
        mocker.patch("uuid.uuid4", return_value=key)
updating the exception: AccessDenied.message to TransferOwnershipAccessDenied.message

* M2 9368 api  update applets in workspace (#1882)

* Update test_workspaces.py

adding the test test_manager_can_update_applet, pending fix

* Update test_workspaces.py

moving the test to the correct place, test_applet

* Update test_applet.py

adding the test test_manager_can_update_applet  to the test_applet file, adding the missing values

* Update test_applet.py

adding 2 new tests:
1. test_coordinator_can_not_update_applet
2. test_editor_can_update_applet
Updating the response for a http status

* Update PR

update the pr after run make cqf command

* fix: Disabling connection pooling (#1916)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Rodrigo Colao Merlo <rodrigo.merlo@metalab.com>
Co-authored-by: Farmer Paul <paul.hh@metalab.com>
Co-authored-by: Marty <mbanting@yahoo.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: SebasYanik <96058252+SebasYanik@users.noreply.github.com>
Co-authored-by: egodoy-metalab <elder.gordy@metalab.com>

Author: 7 people

Pipfile

@@ -14,7 +14,7 @@ azure-storage-blob = "==12.25.*"
 bcrypt = "==4.3.0"
 boto3 = "==1.38.28"
 bytecode = "==0.16.*"
-ddtrace = "==2.21.*"
+ddtrace = "==3.9.*"
 fastapi = "==0.115.*"
 fastapi-mail = "==1.2.9"
 firebase-admin = "==6.8.*"

Pipfile.lock

@@ -84,6 +84,9 @@ pipenv --python /opt/homebrew/bin/python3.13
 | DATABASE\_\_USER                                             | postgres                   | User name for Postgresql Database user                                                                                                                                                                                                                                                                                                 |
 | DATABASE\_\_PASSWORD                                         | postgres                   | Password for Postgresql Database user                                                                                                                                                                                                                                                                                                  |
 | DATABASE\_\_DB                                               | mindlogger_backend         | Database name                                                                                                                                                                                                                                                                                                                          |
+| DATABASE\_\_POOL\_SIZE                                       | 5                          | Database connection pool size                                                                                                                                                                                                                                                                                                          |
+| DATABASE\_\_POOL\_OVERFLOW\_SIZE                             | 10                         | Allowed overflow size of the connection pool                                                                                                                                                                                                                                                                                           |
+| DATABASE\_\_POOL\_TIMEOUT                                    | 30                         | The number of seconds to wait for a connection from the pool to become available                                                                                                                                                                                                                                                       |
 | CORS\_\_ALLOW\_ORIGINS                                       | `*`                        | Represents the list of allowed origins. Set the `Access-Control-Allow-Origin` header. Example: `https://dev.com,http://localohst:8000`                                                                                                                                                                                                 |
 | CORS\_\_ALLOW\_ORIGINS\_REGEX                                | -                          | Regex pattern of allowed origins.                                                                                                                                                                                                                                                                                                      |
 | CORS\_\_ALLOW\_CREDENTIALS                                   | true                       | Set the `Access-Control-Allow-Credentials` header                                                                                                                                                                                                                                                                                      |

README.md

@@ -13,21 +13,11 @@ set -o pipefail
 set -o nounset
 
 
-#NUM_WORKERS=1
-#if command -v nproc >/dev/null 2>&1; then
-#  NUM_WORKERS=$(nproc)
-#  echo "Starting app with $NUM_WORKERS workers..."
-#else
-#  echo "No nproc, defaulting to 1 worker..."
-#fi
-
-NUM_WORKERS=2
-
 # https://www.uvicorn.org/settings/
 export UVICORN_HOST="0.0.0.0"
 export UVICORN_PORT=80
 
 uvicorn main:app \
   --host ${UVICORN_HOST} --port ${UVICORN_PORT} \
   --reload --proxy-headers \
-  --log-config uvicorn_disable_logging.json --workers ${NUM_WORKERS}
+  --log-config uvicorn_disable_logging.json

compose/fastapi/start-backend-datadog

@@ -181,7 +181,7 @@ sidecars:
       DD_LOGS_ENABLED: true
       DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL: false
       DD_CONTAINER_EXCLUDE: "name:datadog name:datadog-agent name:fluent-bit name:firelens_log_router name:ecs-agent name:aws-fargate-supervisor"
-      DD_APM_ENABLED: true
+      DD_APM_ENABLED: false
     secrets:
       DD_API_KEY:
         secretsmanager: "cmiml-feature-${COPILOT_ENVIRONMENT_NAME}:DD_API_KEY::"

copilot/mindlogger-backend/manifest.yml

@@ -22,7 +22,7 @@ class _BaseActivityItemSchema:
 class ActivityItemSchema(_BaseActivityItemSchema, Base):
     __tablename__ = "activity_items"
 
-    activity_id = Column(ForeignKey("activities.id", ondelete="CASCADE"), nullable=False)
+    activity_id = Column(ForeignKey("activities.id", ondelete="CASCADE"), nullable=False, index=True)
 
 
 class ActivityItemHistorySchema(_BaseActivityItemSchema, Base):

src/apps/activities/db/schemas/activity_item.py

@@ -122,8 +122,17 @@ async def create_answer(
         service = AnswerService(session, user.id, answer_session)
         if tz_offset is not None and schema.answer.tz_offset is None:
             schema.answer.tz_offset = tz_offset // 60  # value in minutes
-        async with atomic(answer_session):
-            answer = await service.create_answer(schema, device.device_id if device else None)
+
+        try:
+            async with atomic(answer_session):
+                answer = await service.create_answer(schema, device.device_id if device else None)
+        except Exception as e:
+            logger.error(
+                f"Answer creation failed: applet_id={schema.applet_id}, user_id={user.id}, \
+                    activity_id={schema.activity_id}, error={type(e).__name__}: {e}"
+            )
+            raise
+
         await service.create_report_from_answer(answer)
         if schema.allowed_ehr_ingest:
             await service.trigger_ehr_ingestion(

src/apps/answers/api.py

@@ -57,6 +57,7 @@ class TestApplet:
     history_changes_url = f"{applet_detail_url}/versions/{{version}}/changes"
     applet_base_info_url = f"{applet_detail_url}/base_info"
     access_link_url = f"{applet_detail_url}/access_link"
+    applets_updates_url = f"{applet_list_url}/{{applet_id}}"
 
     public_applet_detail_url = "/public/applets/{key}"
     public_applet_base_info_url = f"{public_applet_detail_url}/base_info"
@@ -1298,3 +1299,70 @@ async def test_applet_retrieve_meta(
         response = await client.get(self.applet_detail_url.format(pk=applet_with_reviewable_activity.id))
         assert response.status_code == http.HTTPStatus.OK
         assert response.json()["appletMeta"]["hasAssessment"]
+
+    async def test_manager_can_update_applet(
+        self, client: TestClient, lucy: User, applet_one_lucy_manager: AppletFull, encryption: Encryption
+    ):
+        client.login(lucy)
+        edit_data = dict(
+            {
+                "display_name": "Updated Name",
+                "encryption": encryption.dict(),
+                "activities": [],
+                "activityFlows": [],
+            }
+        )
+
+        response = await client.put(
+            self.applets_updates_url.format(
+                applet_id=applet_one_lucy_manager.id,
+            ),
+            data=edit_data,
+        )
+        assert response.status_code == http.HTTPStatus.OK, response.json()
+        data = response.json()["result"]
+        assert data["displayName"] == "Updated Name"
+
+    async def test_coordinator_can_not_update_applet(
+        self, client: TestClient, lucy: User, applet_one_lucy_coordinator: AppletFull, encryption: Encryption
+    ):
+        client.login(lucy)
+        edit_data = dict(
+            {
+                "display_name": "Updated Name",
+                "encryption": encryption.dict(),
+                "activities": [],
+                "activityFlows": [],
+            }
+        )
+
+        response = await client.put(
+            self.applets_updates_url.format(
+                applet_id=applet_one_lucy_coordinator.id,
+            ),
+            data=edit_data,
+        )
+        assert response.status_code == http.HTTPStatus.FORBIDDEN, response.json()
+
+    async def test_editor_can_update_applet(
+        self, client: TestClient, lucy: User, applet_one_lucy_editor: AppletFull, encryption: Encryption
+    ):
+        client.login(lucy)
+        edit_data = dict(
+            {
+                "display_name": "Updated Name",
+                "encryption": encryption.dict(),
+                "activities": [],
+                "activityFlows": [],
+            }
+        )
+
+        response = await client.put(
+            self.applets_updates_url.format(
+                applet_id=applet_one_lucy_editor.id,
+            ),
+            data=edit_data,
+        )
+        assert response.status_code == http.HTTPStatus.OK, response.json()
+        data = response.json()["result"]
+        assert data["displayName"] == "Updated Name"

src/apps/applets/tests/test_applet.py

@@ -21,6 +21,7 @@
 from apps.transfer_ownership.crud import TransferCRUD
 from apps.transfer_ownership.errors import TransferEmailError
 from apps.users.domain import User
+from apps.workspaces.errors import TransferOwnershipAccessDenied
 from apps.workspaces.service.user_applet_access import UserAppletAccessService
 
 
@@ -55,6 +56,13 @@ async def applet_one_lucy_respondent(session: AsyncSession, applet_one: AppletFu
     return applet_one
 
 
+@pytest.fixture
+async def applet_one_bob_coordinator_reviewer(session: AsyncSession, applet_one: AppletFull, tom, bob) -> AppletFull:
+    await UserAppletAccessService(session, tom.id, applet_one.id).add_role(bob.id, Role.COORDINATOR)
+    await UserAppletAccessService(session, tom.id, applet_one.id).add_role(bob.id, Role.REVIEWER)
+    return applet_one
+
+
 class TestTransfer(BaseTest):
     fixtures = [
         "transfer_ownership/fixtures/transfers.json",
@@ -536,3 +544,68 @@ async def test_decline_transfer__applet_not_in_transfer(
         result = resp.json()["result"]
         assert len(result) == 1
         assert result[0]["message"] == PermissionsError.message
+
+    async def test_manager_can_not_transfer_ownership(
+        self,
+        client: TestClient,
+        applet_one: AppletFull,
+        lucy: User,
+        tom: User,
+    ):
+        client.login(lucy)
+        data = {"email": tom.email_encrypted}
+        resp = await client.post(self.transfer_url.format(applet_id=applet_one.id), data=data)
+        assert resp.status_code == http.HTTPStatus.FORBIDDEN
+        assert resp.json()["result"][0]["message"] == TransferOwnershipAccessDenied.message
+
+    async def test_respondent_can_not_transfer_ownershipto(
+        self,
+        client: TestClient,
+        applet_one_lucy_respondent: AppletFull,
+        lucy: User,
+        tom: User,
+    ):
+        client.login(lucy)
+        data = {"email": tom.email_encrypted}
+        resp = await client.post(self.transfer_url.format(applet_id=applet_one_lucy_respondent.id), data=data)
+        assert resp.status_code == http.HTTPStatus.FORBIDDEN
+        assert resp.json()["result"][0]["message"] == TransferOwnershipAccessDenied.message
+
+    async def test_reviewer_can_not_transfer_ownership(
+        self,
+        client: TestClient,
+        applet_one_bob_coordinator_reviewer: AppletFull,
+        lucy: User,
+        tom: User,
+    ):
+        client.login(lucy)
+        data = {"email": tom.email_encrypted}
+        resp = await client.post(self.transfer_url.format(applet_id=applet_one_bob_coordinator_reviewer.id), data=data)
+        assert resp.status_code == http.HTTPStatus.FORBIDDEN
+        assert resp.json()["result"][0]["message"] == TransferOwnershipAccessDenied.message
+
+    async def test_editor_can_not_transfer_ownership(
+        self,
+        client: TestClient,
+        applet_one_lucy_editor: AppletFull,
+        lucy: User,
+        tom: User,
+    ):
+        client.login(lucy)
+        data = {"email": tom.email_encrypted}
+        resp = await client.post(self.transfer_url.format(applet_id=applet_one_lucy_editor.id), data=data)
+        assert resp.status_code == http.HTTPStatus.FORBIDDEN
+        assert resp.json()["result"][0]["message"] == TransferOwnershipAccessDenied.message
+
+    async def test_coordinator_can_not_transfer_ownership(
+        self,
+        client: TestClient,
+        applet_one_bob_coordinator_reviewer: AppletFull,
+        lucy: User,
+        tom: User,
+    ):
+        client.login(lucy)
+        data = {"email": tom.email_encrypted}
+        resp = await client.post(self.transfer_url.format(applet_id=applet_one_bob_coordinator_reviewer.id), data=data)
+        assert resp.status_code == http.HTTPStatus.FORBIDDEN
+        assert resp.json()["result"][0]["message"] == TransferOwnershipAccessDenied.message

src/apps/transfer_ownership/tests.py

@@ -7,7 +7,9 @@ class DatabaseSettings(BaseModel):
     password: str = "postgres"
     user: str = "postgres"
     db: str = "mindlogger_backend"
-    pool_size: int = 2
+    pool_size: int = 5
+    pool_overflow_size: int = 10
+    pool_timeout: int = 30
 
     @property
     def url(self) -> str: