Release/2025.06.6 (#1910)

* feat: Enhances OneUp Health integration for document retrieval (M2-8883) (#1879)

* Enhances OneUp Health integration for document retrieval

Improves the OneUp Health integration by adding functionality
to download and store patient documents associated with EHR data.

This change introduces the ability to fetch documents referenced
in DocumentReference resources, store them in a zip file,
and upload the zip to the EHR storage.

It also introduces generic file uploading and listing capabilities
in the EHR storage, along with helper methods.

* Adds metadata to EHR answers

Adds metadata to the EHR answers table to store information about the uploaded zip files, including their names and sizes.

This allows for better tracking and management of the EHR data stored in the system.

* Adds assertion to task ingest user data

Adds an assertion to check that the result of the
`ingest_user_data` function is not None. This ensures that
the task completes successfully and returns a valid result.

* Update src/apps/integrations/oneup_health/service/oneup_health.py

Co-authored-by: Farmer Paul <paul.hh@metalab.com>

* Refactors EHR metadata handling

Updates EHR metadata to use a dedicated data model.

This change introduces `EHRFileMetadata` and `EHRFileTypeEnum`
to provide more structured and type-safe handling of EHR file
metadata, replacing the previous use of dictionaries. This improves
code clarity and maintainability.

* Adds provider name to EHR document filename

Ensures that the EHR document filename includes the
healthcare provider's name (or ID if name unavailable)
to improve identification and organization.

* Adds buckets for answers and operations

Extends MinIO bucket creation to include buckets for answers and operations, in addition to the existing media bucket.
This allows for a more organized storage structure within MinIO and improves separation of concerns.

* Adds a TODO item to optimize zip file creation.

Adds a TODO item to address potential memory issues
when creating zip files, especially when dealing with
large documents or numerous files.

* Refactors EHR file upload process

Streamlines the EHR file upload process by moving the base path
generation logic into the `EHRStorage.upload_file` method.
This change improves code maintainability and reduces redundancy.

---------

Co-authored-by: Farmer Paul <paul.hh@metalab.com>

* fix: Refactors ehr zip download to use storage path (M2-9443) (#1893)

Simplifies the ehr zip download process by directly using the storage path. This eliminates redundant path construction and improves code readability.

* chore: Added worker configuration to uvicorn startup (#1896)

* fix: Optimizes schedule event retrieval for performance (#1897)

* Optimizes event retrieval for schedule app

Improves performance by fetching events and associated
notifications/reminders in bulk using applet IDs.
This reduces database queries and speeds up schedule loading.

* Optimizes event data retrieval

Reduces the number of database queries required to fetch event
notifications and reminders by retrieving them in bulk.

This change improves performance when fetching schedule events,
especially when retrieving events for a large number of users.

* fix: Temp fix for workers (#1902)

* Fix: Enhances EHR exports to filter only allowed subjects answers (M2-9434) (#1894)

* Adjusts retry delay for task processing errors

Modifies the retry mechanism to use a linear delay
after the first failed attempt. This avoids excessive
wait times when processing tasks that encounter errors.

* Filters EHR answers by allowed subjects

Ensures that exported EHR answers are filtered based on the subjects the user is authorized to access.
If a list of allowed subjects is available, it's added as a filter when querying EHR answers.

* Refines subject ID filtering for EHR answers

Ensures that the target subject IDs for EHR answer exports are
filtered to only include IDs that are allowed for the user.
This prevents unauthorized access to data associated with subjects
outside the user's permitted scope.

* Adds reviewer role to EHR data endpoint test

Extends the EHR data endpoint test to include scenarios for
users with the reviewer role. This ensures that reviewers
can properly access and download EHR data relevant to their
responsibilities.

---------

Co-authored-by: Rodrigo Colao Merlo <rodrigo.merlo@metalab.com>
Co-authored-by: Farmer Paul <paul.hh@metalab.com>
Co-authored-by: Marty <mbanting@yahoo.com>

Author: 4 people

compose/minio/create_bucket.sh

@@ -7,11 +7,24 @@
 
 /usr/bin/mc config host add local http://minio:9000 minioaccess miniosecret;
 #/usr/bin/mc rm -r --force local/${CDN__BUCKET};
+#media bucket
 /usr/bin/mc mb -p local/${CDN__BUCKET};
 /usr/bin/mc policy set download local/${CDN__BUCKET};
 /usr/bin/mc policy set public local/${CDN__BUCKET};
 /usr/bin/mc anonymous set upload local/${CDN__BUCKET};
 /usr/bin/mc anonymous set download local/${CDN__BUCKET};
 /usr/bin/mc anonymous set public local/${CDN__BUCKET};
+#answer bucket
+/usr/bin/mc mb -p local/${CDN__BUCKET_ANSWER};
+/usr/bin/mc policy set download local/${CDN__BUCKET_ANSWER};
+/usr/bin/mc policy set public local/${CDN__BUCKET_ANSWER};
+/usr/bin/mc anonymous set upload local/${CDN__BUCKET_ANSWER};
+/usr/bin/mc anonymous set download local/${CDN__BUCKET_ANSWER};
+#operations bucket
+/usr/bin/mc mb -p local/${CDN__BUCKET_OPERATIONS};
+/usr/bin/mc policy set download local/${CDN__BUCKET_OPERATIONS};
+/usr/bin/mc policy set public local/${CDN__BUCKET_OPERATIONS};
+/usr/bin/mc anonymous set upload local/${CDN__BUCKET_OPERATIONS};
+/usr/bin/mc anonymous set download local/${CDN__BUCKET_OPERATIONS};
 
-exit 0;
+exit 0;

docker-compose.yaml

@@ -118,7 +118,9 @@ services:
     image: minio/mc
     container_name: mindlogger_minio_mc
     environment:
-      CDN__BUCKET: "${CDN__BUCKET:-media}"
+      CDN__BUCKET: "${CDN__BUCKET:-cmi-media-local}"
+      CDN__BUCKET_ANSWER: "${CDN__BUCKET_ANSWER:-cmi-answer-local}"
+      CDN__BUCKET_OPERATIONS: "${CDN__BUCKET_OPERATIONS:-cmi-ops-local}"
     depends_on:
       - minio
     volumes:

src/apps/answers/db/schemas.py

@@ -107,3 +107,4 @@ class AnswerEHRSchema(Base):
     activity_id = Column(UUID(as_uuid=True), index=True)
     ehr_storage_uri = Column(Text())
     ehr_ingestion_status = Column(Text())
+    meta = Column(JSONB())

src/apps/answers/domain/answers.py

@@ -18,6 +18,7 @@
 from apps.activity_flows.domain.flow_full import FlowFull, FlowHistoryWithActivityFlat, FlowHistoryWithActivityFull
 from apps.answers.domain.answer_items import AnswerItem, ItemAnswerCreate
 from apps.applets.domain.base import AppletBaseInfo
+from apps.integrations.oneup_health.service.domain import EHRMetadata
 from apps.integrations.prolific.domain import ProlificParamsActivityAnswer
 from apps.shared.domain import InternalModel, PublicModel, Response
 from apps.shared.domain.custom_validations import datetime_from_ms
@@ -748,6 +749,7 @@ class AnswerEHR(InternalModel):
     ehr_ingestion_status: EHRIngestionStatus
     activity_id: uuid.UUID
     ehr_storage_uri: str | None
+    meta: EHRMetadata | None
 
 
 class AnswerEHRFull(AnswerEHR):

src/apps/answers/service.py

@@ -2049,7 +2049,16 @@ async def get_submissions_metadata_by_subject(self, subject_id: uuid.UUID) -> Su
         return submissions_activity_metadata
 
     async def export_ehr_answers(self, applet_id: uuid.UUID, query_params: QueryParams) -> list[AnswerEHRFull]:
-        ehr_answers = await AnswersEHRCRUD(self.answer_session).export_ehr_answers(applet_id, **query_params.filters)
+        allowed_subjects = await self._get_allowed_subjects(applet_id)
+
+        filters = query_params.filters
+        if allowed_subjects is not None:
+            if target_subject_ids := filters.get("target_subject_ids"):
+                filters["target_subject_ids"] = list(set(allowed_subjects).intersection(set(target_subject_ids)))
+            else:
+                filters["target_subject_ids"] = allowed_subjects
+
+        ehr_answers = await AnswersEHRCRUD(self.answer_session).export_ehr_answers(applet_id, **filters)
 
         return [AnswerEHRFull(**ehr_answer) for ehr_answer in ehr_answers]
 

src/apps/answers/tests/test_answers.py

@@ -114,7 +114,9 @@ async def applet_one_sam_subject(session: AsyncSession, applet_one: AppletFull,
 
 
 @pytest.fixture
-async def bob_reviewer_in_applet_with_reviewable_activity(session, tom, bob, applet_with_reviewable_activity) -> User:
+async def bob_reviewer_in_applet_with_reviewable_activity(
+    session: AsyncSession, tom, bob, applet_with_reviewable_activity
+) -> User:
     tom_subject = await SubjectsService(session, tom.id).get_by_user_and_applet(
         tom.id, applet_with_reviewable_activity.id
     )
@@ -281,6 +283,35 @@ async def tom_answer_on_reviewable_applet(
     )
 
 
+@pytest.fixture
+async def lucy_answer_on_reviewable_applet(
+    session: AsyncSession,
+    lucy_manager_in_applet_with_reviewable_activity: User,
+    applet_with_reviewable_activity: AppletFull,
+) -> AnswerSchema:
+    await UserAppletAccessService(
+        session, lucy_manager_in_applet_with_reviewable_activity.id, applet_with_reviewable_activity.id
+    ).add_role(lucy_manager_in_applet_with_reviewable_activity.id, Role.RESPONDENT)
+
+    answer_service = AnswerService(session, lucy_manager_in_applet_with_reviewable_activity.id)
+    return await answer_service.create_answer(
+        AppletAnswerCreate(
+            applet_id=applet_with_reviewable_activity.id,
+            version=applet_with_reviewable_activity.version,
+            submit_id=uuid.uuid4(),
+            activity_id=applet_with_reviewable_activity.activities[0].id,
+            answer=ItemAnswerCreate(
+                item_ids=[applet_with_reviewable_activity.activities[0].items[0].id],
+                start_time=datetime.datetime.now(datetime.UTC),
+                end_time=datetime.datetime.now(datetime.UTC),
+                user_public_key=str(lucy_manager_in_applet_with_reviewable_activity.id),
+            ),
+            client=ClientMeta(app_id=f"{uuid.uuid4()}", app_version="1.1", width=984, height=623),
+            consent_to_share=False,
+        )
+    )
+
+
 @pytest.fixture
 async def lucy_answer(session: AsyncSession, lucy: User, applet: AppletFull) -> AnswerSchema:
     answer_service = AnswerService(session, lucy.id)
@@ -4518,6 +4549,83 @@ def _mock_download_ehr_zip(storage_path, data, file_buffer):
                 assert file_list == file_names
                 assert str(tom_answer_activity_flow.submit_id) in file_names[0]
 
+    @pytest.mark.parametrize(
+        "user_fixture,role",
+        (
+            ("tom", Role.OWNER),
+            ("bob", Role.REVIEWER),
+        ),
+    )
+    async def test_applet_ehr_data_endpoint_reviewer(
+        self,
+        client,
+        session,
+        tom,
+        bob_reviewer_in_applet_with_reviewable_activity,
+        lucy_answer_on_reviewable_applet,
+        tom_answer_on_reviewable_applet,
+        user_fixture,
+        role,
+        request,
+    ):
+        # Create EHR record
+        answer_ehr = AnswerEHR(
+            submit_id=lucy_answer_on_reviewable_applet.submit_id,
+            ehr_ingestion_status=EHRIngestionStatus.COMPLETED,
+            ehr_storage_uri="fake/ehr/storage/uri",
+            activity_id=uuid.UUID(lucy_answer_on_reviewable_applet.activity_history_id.split("_")[0]),
+        )
+        await AnswersEHRCRUD(session=session).upsert(answer_ehr)
+
+        # Create EHR record
+        answer_ehr = AnswerEHR(
+            submit_id=tom_answer_on_reviewable_applet.submit_id,
+            ehr_ingestion_status=EHRIngestionStatus.COMPLETED,
+            ehr_storage_uri="tom/fake/ehr/storage/uri",
+            activity_id=uuid.UUID(tom_answer_on_reviewable_applet.activity_history_id.split("_")[0]),
+        )
+        await AnswersEHRCRUD(session=session).upsert(answer_ehr)
+
+        file_names = []
+
+        def _mock_download_ehr_zip(storage_path, data, file_buffer):
+            file_buffer.write(b"mocked EHR zip data")
+            file_buffer.seek(0)
+
+            file_name = EHRStorage.ehr_zip_filename(data)
+            file_names.append(file_name)
+
+            return file_name
+
+        with patch(
+            "apps.integrations.oneup_health.service.ehr_storage.EHRStorage.download_ehr_zip"
+        ) as download_ehr_zip:
+            download_ehr_zip.side_effect = _mock_download_ehr_zip
+
+            login_user = request.getfixturevalue(user_fixture)
+            client.login(login_user)
+
+            # Request EHR data
+            response = await client.get(
+                self.applet_ehr_answers_export_url.format(applet_id=lucy_answer_on_reviewable_applet.applet_id)
+            )
+            assert response.status_code == 200
+            assert response.headers["Content-Type"] == "application/zip"
+            assert response.headers["Content-Disposition"] == "attachment; filename=EHR.zip"
+
+            response_body = response.read()
+            with zipfile.ZipFile(io.BytesIO(response_body), "r") as zip_file:
+                file_list = zip_file.namelist()
+                if role == Role.REVIEWER:
+                    assert len(file_list) == 1
+                    assert file_list == file_names
+                    assert str(tom_answer_on_reviewable_applet.submit_id) in file_names[0]
+                else:
+                    assert len(file_list) == 2
+                    assert file_list == file_names
+                    assert str(tom_answer_on_reviewable_applet.submit_id) in file_names[0]
+                    assert str(lucy_answer_on_reviewable_applet.submit_id) in file_names[1]
+
     @pytest.mark.asyncio
     async def test_applet_ehr_data_endpoint_without_ehr(self, client, session, tom, tom_answer):
         # Create EHR record

src/apps/integrations/oneup_health/service/domain.py

@@ -1,5 +1,6 @@
 import uuid
 from datetime import datetime
+from enum import StrEnum
 
 from pydantic import Field
 
@@ -15,3 +16,19 @@ class EHRData(InternalModel):
     activity_id: uuid.UUID
     target_subject_id: uuid.UUID
     user_id: uuid.UUID
+
+
+class EHRFileTypeEnum(StrEnum):
+    DOCS = "DOCS"
+    EHR = "EHR"
+
+
+class EHRFileMetadata(InternalModel):
+    name: str
+    size: int
+    type: EHRFileTypeEnum
+
+
+class EHRMetadata(InternalModel):
+    zip_files: list[EHRFileMetadata]
+    storage_path: str

src/apps/integrations/oneup_health/service/ehr_storage.py

@@ -27,6 +27,19 @@ def ehr_zip_filename(data: EHRData) -> str:
         )
         return filename
 
+    @staticmethod
+    def docs_zip_filename(data: EHRData) -> str:
+        provider_name = (
+            slugify(data.healthcare_provider_name, separator="-")
+            if data.healthcare_provider_name
+            else data.healthcare_provider_id
+        )
+        filename = (
+            f"{data.target_subject_id}_{data.activity_id}_"
+            f"{data.submit_id}_{data.date.strftime('%Y%m%d')}_{provider_name}_DOCS.zip"
+        )
+        return filename
+
     def _get_storage_path(self, base_path: str, key: str) -> str:
         index = key.find(base_path)
         if index == -1:  # substring not found
@@ -59,7 +72,18 @@ async def upload_resources(self, data: EHRData) -> tuple[str, str]:
 
         return self._get_storage_path(base_path, key), filename
 
-    async def upload_ehr_zip(self, resources_files: list[str], data: EHRData) -> str:
+    async def upload_file(self, data: EHRData, filename: str, content: bytes) -> None:
+        base_path = self._get_base_path(data)
+        key = self._cdn_client.generate_key(FileScopeEnum.EHR, base_path, filename)
+
+        file_buffer = io.BytesIO(content)
+        file_buffer.seek(0)
+
+        await self._cdn_client.upload(key, file_buffer)
+
+        file_buffer.close()
+
+    async def upload_ehr_zip(self, resources_files: list[str], data: EHRData) -> tuple[str, int]:
         base_path = self._get_base_path(data)
         filename = EHRStorage.ehr_zip_filename(data)
         key = self._cdn_client.generate_key(FileScopeEnum.EHR, base_path, filename)
@@ -80,8 +104,9 @@ async def upload_ehr_zip(self, resources_files: list[str], data: EHRData) -> str
                     file_buffer.close()
 
             zip_buffer.seek(0)
+            file_size = zip_buffer.getbuffer().nbytes
             await self._cdn_client.upload(key, zip_buffer)
-            return key
+            return filename, file_size
         finally:
             zip_buffer.close()