Optimizes schedule event retrieval for performance - #1897
Merged
Conversation
Improves performance by fetching events and associated notifications/reminders in bulk using applet IDs. This reduces database queries and speeds up schedule loading.
|
➡️ Preview environment failed to be destroyed |
Reduces the number of database queries required to fetch event notifications and reminders by retrieving them in bulk. This change improves performance when fetching schedule events, especially when retrieving events for a large number of users.
mbanting
marked this pull request as draft
June 25, 2025 18:44
mbanting
marked this pull request as ready for review
June 25, 2025 18:47
SebasYanik
self-requested a review
June 25, 2025 18:47
farmerpaul
self-requested a review
June 25, 2025 19:01
farmerpaul
approved these changes
Jun 25, 2025
farmerpaul
left a comment
farmerpaul
left a comment
Contributor
There was a problem hiding this comment.
Changes look good to me, and Claude 4 agrees: the changes only improve database performance and do not introduce any regressions. (And it even provided a handy analysis.) Nice work @rcmerlo!
Contributor
|
This looks good overall and the batching should definitely help with the CPU spikes we have been seeing. A couple of questions. Do we have a sense of the upper bound on the number of events that could be loaded into memory at once, and are there any edge cases where the notification or reminder maps could get out of sync? Otherwise, I think this is a solid improvement. Let's monitor after deploy and see how it performs under load. |
SebasYanik
approved these changes
Jun 25, 2025
Contributor
|
Here's the top twenty applets by # of events... (assuming events are just records from the DB table "events"). This isn't exactly an upper bound, as users can theoretically belong to multiple applets. But this gives us an idea that "thousands" is in not an unreasonable number to expect. I can't imagine what happens whey users of these applets hit the current endpoint...1800 queries! Interestingly, there are only 20k records in the entire events table. 62be21d7-cd01-4b9b-975a-39750d940f59 NIMH Rhythms and Mood Family Study EMA 1841 |
sultanofcardio
approved these changes
Jun 25, 2025
sultanofcardio
left a comment
sultanofcardio
left a comment
Contributor
There was a problem hiding this comment.
I think this change is great, and we may see the biggest return from the optimisation of the get_events_by_user_and_applet function. Nice work @rcmerlo
aweiland
added a commit
that referenced
this pull request
Jul 7, 2025
* feat: Enhances OneUp Health integration for document retrieval (M2-8883) (#1879) * Enhances OneUp Health integration for document retrieval Improves the OneUp Health integration by adding functionality to download and store patient documents associated with EHR data. This change introduces the ability to fetch documents referenced in DocumentReference resources, store them in a zip file, and upload the zip to the EHR storage. It also introduces generic file uploading and listing capabilities in the EHR storage, along with helper methods. * Adds metadata to EHR answers Adds metadata to the EHR answers table to store information about the uploaded zip files, including their names and sizes. This allows for better tracking and management of the EHR data stored in the system. * Adds assertion to task ingest user data Adds an assertion to check that the result of the `ingest_user_data` function is not None. This ensures that the task completes successfully and returns a valid result. * Update src/apps/integrations/oneup_health/service/oneup_health.py Co-authored-by: Farmer Paul <paul.hh@metalab.com> * Refactors EHR metadata handling Updates EHR metadata to use a dedicated data model. This change introduces `EHRFileMetadata` and `EHRFileTypeEnum` to provide more structured and type-safe handling of EHR file metadata, replacing the previous use of dictionaries. This improves code clarity and maintainability. * Adds provider name to EHR document filename Ensures that the EHR document filename includes the healthcare provider's name (or ID if name unavailable) to improve identification and organization. * Adds buckets for answers and operations Extends MinIO bucket creation to include buckets for answers and operations, in addition to the existing media bucket. This allows for a more organized storage structure within MinIO and improves separation of concerns. * Adds a TODO item to optimize zip file creation. Adds a TODO item to address potential memory issues when creating zip files, especially when dealing with large documents or numerous files. * Refactors EHR file upload process Streamlines the EHR file upload process by moving the base path generation logic into the `EHRStorage.upload_file` method. This change improves code maintainability and reduces redundancy. --------- Co-authored-by: Farmer Paul <paul.hh@metalab.com> * fix: Refactors ehr zip download to use storage path (M2-9443) (#1893) Simplifies the ehr zip download process by directly using the storage path. This eliminates redundant path construction and improves code readability. * chore: Added worker configuration to uvicorn startup (#1896) * fix: Optimizes schedule event retrieval for performance (#1897) * Optimizes event retrieval for schedule app Improves performance by fetching events and associated notifications/reminders in bulk using applet IDs. This reduces database queries and speeds up schedule loading. * Optimizes event data retrieval Reduces the number of database queries required to fetch event notifications and reminders by retrieving them in bulk. This change improves performance when fetching schedule events, especially when retrieving events for a large number of users. * fix: Temp fix for workers (#1902) * Fix: Enhances EHR exports to filter only allowed subjects answers (M2-9434) (#1894) * Adjusts retry delay for task processing errors Modifies the retry mechanism to use a linear delay after the first failed attempt. This avoids excessive wait times when processing tasks that encounter errors. * Filters EHR answers by allowed subjects Ensures that exported EHR answers are filtered based on the subjects the user is authorized to access. If a list of allowed subjects is available, it's added as a filter when querying EHR answers. * Refines subject ID filtering for EHR answers Ensures that the target subject IDs for EHR answer exports are filtered to only include IDs that are allowed for the user. This prevents unauthorized access to data associated with subjects outside the user's permitted scope. * Adds reviewer role to EHR data endpoint test Extends the EHR data endpoint test to include scenarios for users with the reviewer role. This ensures that reviewers can properly access and download EHR data relevant to their responsibilities. --------- Co-authored-by: Rodrigo Colao Merlo <rodrigo.merlo@metalab.com> Co-authored-by: Farmer Paul <paul.hh@metalab.com> Co-authored-by: Marty <mbanting@yahoo.com>
aweiland
added a commit
that referenced
this pull request
Jul 14, 2025
* chore: Add script to delete specific CAMHI applets from library (M2-8175) * fix: Relax validations during answer submission (M2-8924) (#1791) This PR removes the device ID and event history ID validations in the answer submission endpoint. This should make it possible to submits answers again, even when having invalid values for these parameters * fix: Prevent upsert query if events list is empty (M2-8924) (#1792) This PR adds an if statement that prevents attempted insertion to the user_device_events_history table when the user has no events. This can happen for a few reasons, mainly: The user has been deleted, but they still have a valid auth token and consume this endpoint via the mobile app The user is not part of any applets * fix: Temp fix for workers (#1902) * Release/2025.06.6 (#1910) * feat: Enhances OneUp Health integration for document retrieval (M2-8883) (#1879) * Enhances OneUp Health integration for document retrieval Improves the OneUp Health integration by adding functionality to download and store patient documents associated with EHR data. This change introduces the ability to fetch documents referenced in DocumentReference resources, store them in a zip file, and upload the zip to the EHR storage. It also introduces generic file uploading and listing capabilities in the EHR storage, along with helper methods. * Adds metadata to EHR answers Adds metadata to the EHR answers table to store information about the uploaded zip files, including their names and sizes. This allows for better tracking and management of the EHR data stored in the system. * Adds assertion to task ingest user data Adds an assertion to check that the result of the `ingest_user_data` function is not None. This ensures that the task completes successfully and returns a valid result. * Update src/apps/integrations/oneup_health/service/oneup_health.py Co-authored-by: Farmer Paul <paul.hh@metalab.com> * Refactors EHR metadata handling Updates EHR metadata to use a dedicated data model. This change introduces `EHRFileMetadata` and `EHRFileTypeEnum` to provide more structured and type-safe handling of EHR file metadata, replacing the previous use of dictionaries. This improves code clarity and maintainability. * Adds provider name to EHR document filename Ensures that the EHR document filename includes the healthcare provider's name (or ID if name unavailable) to improve identification and organization. * Adds buckets for answers and operations Extends MinIO bucket creation to include buckets for answers and operations, in addition to the existing media bucket. This allows for a more organized storage structure within MinIO and improves separation of concerns. * Adds a TODO item to optimize zip file creation. Adds a TODO item to address potential memory issues when creating zip files, especially when dealing with large documents or numerous files. * Refactors EHR file upload process Streamlines the EHR file upload process by moving the base path generation logic into the `EHRStorage.upload_file` method. This change improves code maintainability and reduces redundancy. --------- Co-authored-by: Farmer Paul <paul.hh@metalab.com> * fix: Refactors ehr zip download to use storage path (M2-9443) (#1893) Simplifies the ehr zip download process by directly using the storage path. This eliminates redundant path construction and improves code readability. * chore: Added worker configuration to uvicorn startup (#1896) * fix: Optimizes schedule event retrieval for performance (#1897) * Optimizes event retrieval for schedule app Improves performance by fetching events and associated notifications/reminders in bulk using applet IDs. This reduces database queries and speeds up schedule loading. * Optimizes event data retrieval Reduces the number of database queries required to fetch event notifications and reminders by retrieving them in bulk. This change improves performance when fetching schedule events, especially when retrieving events for a large number of users. * fix: Temp fix for workers (#1902) * Fix: Enhances EHR exports to filter only allowed subjects answers (M2-9434) (#1894) * Adjusts retry delay for task processing errors Modifies the retry mechanism to use a linear delay after the first failed attempt. This avoids excessive wait times when processing tasks that encounter errors. * Filters EHR answers by allowed subjects Ensures that exported EHR answers are filtered based on the subjects the user is authorized to access. If a list of allowed subjects is available, it's added as a filter when querying EHR answers. * Refines subject ID filtering for EHR answers Ensures that the target subject IDs for EHR answer exports are filtered to only include IDs that are allowed for the user. This prevents unauthorized access to data associated with subjects outside the user's permitted scope. * Adds reviewer role to EHR data endpoint test Extends the EHR data endpoint test to include scenarios for users with the reviewer role. This ensures that reviewers can properly access and download EHR data relevant to their responsibilities. --------- Co-authored-by: Rodrigo Colao Merlo <rodrigo.merlo@metalab.com> Co-authored-by: Farmer Paul <paul.hh@metalab.com> Co-authored-by: Marty <mbanting@yahoo.com> * fix: Disabling connection pooling (#1916) --------- Co-authored-by: Carlos Chacon <carlos.chacon@metalab.com> Co-authored-by: Marty <mbanting@yahoo.com> Co-authored-by: Kenroy Gobourne <14842108+sultanofcardio@users.noreply.github.com> Co-authored-by: Rodrigo Colao Merlo <rodrigo.merlo@metalab.com> Co-authored-by: Farmer Paul <paul.hh@metalab.com>
aweiland
added a commit
that referenced
this pull request
Jul 14, 2025
* feat: Enhances OneUp Health integration for document retrieval (M2-8883) (#1879) * Enhances OneUp Health integration for document retrieval Improves the OneUp Health integration by adding functionality to download and store patient documents associated with EHR data. This change introduces the ability to fetch documents referenced in DocumentReference resources, store them in a zip file, and upload the zip to the EHR storage. It also introduces generic file uploading and listing capabilities in the EHR storage, along with helper methods. * Adds metadata to EHR answers Adds metadata to the EHR answers table to store information about the uploaded zip files, including their names and sizes. This allows for better tracking and management of the EHR data stored in the system. * Adds assertion to task ingest user data Adds an assertion to check that the result of the `ingest_user_data` function is not None. This ensures that the task completes successfully and returns a valid result. * Update src/apps/integrations/oneup_health/service/oneup_health.py Co-authored-by: Farmer Paul <paul.hh@metalab.com> * Refactors EHR metadata handling Updates EHR metadata to use a dedicated data model. This change introduces `EHRFileMetadata` and `EHRFileTypeEnum` to provide more structured and type-safe handling of EHR file metadata, replacing the previous use of dictionaries. This improves code clarity and maintainability. * Adds provider name to EHR document filename Ensures that the EHR document filename includes the healthcare provider's name (or ID if name unavailable) to improve identification and organization. * Adds buckets for answers and operations Extends MinIO bucket creation to include buckets for answers and operations, in addition to the existing media bucket. This allows for a more organized storage structure within MinIO and improves separation of concerns. * Adds a TODO item to optimize zip file creation. Adds a TODO item to address potential memory issues when creating zip files, especially when dealing with large documents or numerous files. * Refactors EHR file upload process Streamlines the EHR file upload process by moving the base path generation logic into the `EHRStorage.upload_file` method. This change improves code maintainability and reduces redundancy. --------- Co-authored-by: Farmer Paul <paul.hh@metalab.com> * fix: Refactors ehr zip download to use storage path (M2-9443) (#1893) Simplifies the ehr zip download process by directly using the storage path. This eliminates redundant path construction and improves code readability. * chore: Added worker configuration to uvicorn startup (#1896) * fix: Optimizes schedule event retrieval for performance (#1897) * Optimizes event retrieval for schedule app Improves performance by fetching events and associated notifications/reminders in bulk using applet IDs. This reduces database queries and speeds up schedule loading. * Optimizes event data retrieval Reduces the number of database queries required to fetch event notifications and reminders by retrieving them in bulk. This change improves performance when fetching schedule events, especially when retrieving events for a large number of users. * fix: Temp fix for workers (#1902) * Fix: Enhances EHR exports to filter only allowed subjects answers (M2-9434) (#1894) * Adjusts retry delay for task processing errors Modifies the retry mechanism to use a linear delay after the first failed attempt. This avoids excessive wait times when processing tasks that encounter errors. * Filters EHR answers by allowed subjects Ensures that exported EHR answers are filtered based on the subjects the user is authorized to access. If a list of allowed subjects is available, it's added as a filter when querying EHR answers. * Refines subject ID filtering for EHR answers Ensures that the target subject IDs for EHR answer exports are filtered to only include IDs that are allowed for the user. This prevents unauthorized access to data associated with subjects outside the user's permitted scope. * Adds reviewer role to EHR data endpoint test Extends the EHR data endpoint test to include scenarios for users with the reviewer role. This ensures that reviewers can properly access and download EHR data relevant to their responsibilities. * Disabling APM for feature environments * fix: Adds index to activity_items.activity_id (#1901) Improves query performance by adding an index to the activity_id column in the activity_items table. Optimizes database lookups and enhances the efficiency of related queries. * [pip] Dependabot: Bump ddtrace from 2.21.8 to 3.9.4 (#1908) Bumps [ddtrace](https://github.com/DataDog/dd-trace-py) from 2.21.8 to 3.9.4. - [Release notes](https://github.com/DataDog/dd-trace-py/releases) - [Changelog](https://github.com/DataDog/dd-trace-py/blob/main/CHANGELOG.md) - [Commits](DataDog/dd-trace-py@v2.21.8...v3.9.4) --- updated-dependencies: - dependency-name: ddtrace dependency-version: 3.9.4 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [pip] Dependabot: Bump botocore from 1.38.28 to 1.38.46 (#1907) --- updated-dependencies: - dependency-name: botocore dependency-version: 1.38.46 dependency-type: indirect update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [pip] Dependabot: Bump google-api-python-client from 2.170.0 to 2.174.0 (#1906) --- updated-dependencies: - dependency-name: google-api-python-client dependency-version: 2.174.0 dependency-type: indirect update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Use the real exception message (#1904) * chore: Removing workers (#1909) * chore: Enable Database connection pooling (M2-9464) (#1898) * chore: Enabled connection pooling * fixed timeout and docs * cq * feat: Add logging around /answers 422 errors (M2-9475) (#1905) * feat: Add logging around /answers 422 errors (M2-9475) * Remove unused methods * M2 9399 transfer ownership (#1888) * Update errors.py Adding the expected message error for the transfer ownvership test * Update tests.py Adding the new migrated test from TAF: - Adding the fixture applet_one_bob_coordinator_reviewer - Adding the tests test_manager_can_not_transfer_ownership test_respondent_can_not_transfer_ownershipto test_reviewer_can_not_transfer_ownership test_editor_can_not_transfer_ownership test_coordinator_can_not_transfer_ownership * Update errors.py deleting the error exception to use TransferOwnershipAccessDenied * Update tests.py deleting in all the test key = uuid.uuid4() mocker.patch("uuid.uuid4", return_value=key) updating the exception: AccessDenied.message to TransferOwnershipAccessDenied.message * M2 9368 api update applets in workspace (#1882) * Update test_workspaces.py adding the test test_manager_can_update_applet, pending fix * Update test_workspaces.py moving the test to the correct place, test_applet * Update test_applet.py adding the test test_manager_can_update_applet to the test_applet file, adding the missing values * Update test_applet.py adding 2 new tests: 1. test_coordinator_can_not_update_applet 2. test_editor_can_update_applet Updating the response for a http status * Update PR update the pr after run make cqf command * fix: Disabling connection pooling (#1916) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Rodrigo Colao Merlo <rodrigo.merlo@metalab.com> Co-authored-by: Farmer Paul <paul.hh@metalab.com> Co-authored-by: Marty <mbanting@yahoo.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: SebasYanik <96058252+SebasYanik@users.noreply.github.com> Co-authored-by: egodoy-metalab <elder.gordy@metalab.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Improves schedule performance by optimizing event retrieval:
🔗 Jira Ticket M2-#