Conversation
- README.md: Update from Node 14/16 to Node 20+ (v22, v24 supported) - README.md: Update MongoDB from 4.2/4.4 to 4.4+ (5.0, 6.0 supported) - CONTRIBUTING.md: Add 'Running Tests Locally' section - Document NODE_ENV=test requirement for safety - Document npm scripts: test:unit, test:integration Addresses DOC-NODE-001 and DOC-ENV-001 from release-15.0.7-documentation.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- docs/data-schemas/treatments-schema.md: Document REQ-SYNC-072 identifier normalization - Describe how Loop/AAPS/xDrip+ sync fields are unified into identifier - Add example showing UUID _id → identifier promotion - Update sync/reconciliation fields table - CHANGELOG.md: Create changelog for 15.0.7 release - Document UUID/identifier handling (REQ-SYNC-072) - Document test infrastructure improvements (GAP-SYNC-046) - Document Node.js/MongoDB requirement changes - Document bug fixes and documentation updates Addresses DOC-API-001 and DOC-CHANGELOG from release-15.0.7-documentation.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- env.js: Add UUID_HANDLING env var (default: false)
- query.js: Add UUID detection in normalizeIdValue()
- When UUID_HANDLING=true and _id is UUID, search by identifier field
- Returns searchByIdentifier flag to redirect query
- treatments.js: Move queryOpts inside query_for() for env access
- entries.js: Same pattern for entries collection
When UUID_HANDLING=true:
- GET /treatments/{uuid} searches by identifier field
- DELETE /treatments/{uuid} deletes by identifier field
- Same behavior for entries collection
When UUID_HANDLING=false (default):
- UUID _id values return empty results (safe, no crash)
- Maintains backwards compatibility
Refs: uuid-feature-flag, uuid-query-impl from uuid-identifier-lookup.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace Date.now() with the pre-captured 'now' variable in the 'set a pill to BWP with infos' test. This prevents timing drift between when test data timestamps are set and when the sandbox is initialized, eliminating flaky failures in CI environments. Refs: BWP-TIME-001, GAP-TEST-001 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add 6 tests verifying UUID_HANDLING env var behavior: - UUID-OFF-001: GET by UUID returns empty (no crash) - UUID-OFF-002: DELETE by UUID deletes nothing (no crash) - UUID-ON-001: GET by UUID finds treatment via identifier - UUID-ON-002: DELETE by UUID removes treatment via identifier - UUID-ON-003: ObjectId still works normally - UUID-ON-004: Non-matching UUID returns empty Tests use clearModuleCache() to reload env.js with different flag values. Refs: uuid-test-flag-off, uuid-test-flag-on, REQ-SYNC-072 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add 7 edge case tests for UUID_HANDLING: - UUID-EDGE-001: 23-char hex (invalid ObjectId) - UUID-EDGE-002: 25-char hex (too long) - UUID-EDGE-003: UUID without hyphens not recognized - UUID-EDGE-004: Empty _id query - UUID-EDGE-005: Same identifier upsert behavior - UUID-EDGE-006: Uppercase UUID matching - UUID-EDGE-007: Valid ObjectId still works Refs: uuid-test-edge, REQ-SYNC-072 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document MongoDB pool settings and AUTH_FAIL_DELAY for test tuning. Useful for CI or resource-constrained environments. Refs: DOC-ENV-002 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document entries collection fields, sysTime+type dedup behavior, and identifier normalization for Trio/xDrip+ UUID _id uploads. Refs: DOC-API-002, REQ-SYNC-072, GAP-SYNC-045 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
API returns _id as plain string "507f1f77bcf86cd799439011", not
MongoDB Extended JSON format {"$oid": "..."}.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Update default from false to true to enable AID client compatibility
out of the box:
- lib/server/env.js: readENVTruthy('UUID_HANDLING', true)
- README.md: Document UUID_HANDLING in Features section
- docs/example-template.env: Update comments, show true as default
Rationale:
- Loop, Trio, AAPS, xDrip+ use UUID sync patterns by default
- Before MongoDB 5.x, UUID _id didn't crash (just didn't CRUD properly)
- ObjectID users completely unaffected (quirk only triggers on UUID)
- Can set UUID_HANDLING=false for strict mode if needed
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
README.md: - Add MONGO_POOL_SIZE, MONGO_MIN_POOL_SIZE, MONGO_MAX_IDLE_TIME_MS to Core section for production tuning CONTRIBUTING.md: - Add 'Advanced Test Scripts' section documenting: - test:stress for concurrent write testing - test:flaky variants for detecting flaky tests - test:timing for finding slow tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Avoid implying all AID clients use the same pattern or that any specific implementation is incorrect. Different clients have divergent sync patterns - the feature accommodates this variety. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Actual defaults in lib/storage/mongo-storage.js: - MONGO_POOL_SIZE: 5 (not 10) - MONGO_MIN_POOL_SIZE: 0 (not 1) - MONGO_MAX_IDLE_TIME_MS: 30000 (not 10000) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The feature only handles the specific case where a UUID is sent as the _id field itself. It does NOT affect: - AAPS (uses 'identifier' field) - xDrip+ (uses 'uuid' field) - Loop carbs/doses (uses 'syncIdentifier' field) Only affects: - Loop overrides (_id: syncIdentifier.uuidString) - Trio CGM entries (_id: UUID) See docs/10-domain/client-id-handling-deep-dive.md for full analysis. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Write normalization (syncIdentifier/uuid → identifier) is ALWAYS on - UUID_HANDLING flag only controls READ path (GET/DELETE by UUID) - Fix default: UUID_HANDLING=true (not false) - Remove incorrect xDrip+ mention from entries (doesn't use UUID _id) - Clarify that flag only affects API calls with UUID as _id parameter Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
UUID_HANDLING default changed to true in 15.0.7. Test now explicitly sets UUID_HANDLING=false rather than deleting the env var. 742 passing, 1 pending. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
REQ-SYNC-072 scope correction: normalizeTreatmentId() should ONLY handle UUID values in the _id field, not copy syncIdentifier or uuid fields to identifier. Changes: - normalizeTreatmentId(): Only extract UUID from _id to identifier - normalizeEntryId(): Same fix for entries collection - upsertQueryFor(): Add syncIdentifier and uuid as dedup fallbacks (fields are preserved, not copied to identifier) - Batch POST: Fetch _id for docs deduped by syncIdentifier/uuid Test updates: - TEST-ID-003, TEST-V1-ID-004: Updated to expect identifier NOT copied from syncIdentifier (scope fix) Affected clients: - Loop overrides (UUID _id → identifier): Still works - Loop carbs/doses (syncIdentifier): Dedup works, no identifier copy - xDrip+ (uuid): Dedup works, no identifier copy - AAPS (identifier): Unchanged Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
syncIdentifier and uuid fields are used for dedup, not copied to identifier. Only UUID values in _id field are extracted to identifier. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
UUID_HANDLING should ONLY affect UUID values in the _id field. Previous commit incorrectly added server-side dedup for syncIdentifier and uuid fields, which was never part of the original behavior. Changes: - upsertQueryFor(): Remove syncIdentifier/uuid as dedup keys - Batch POST: Only fetch existing IDs by identifier, not by syncIdentifier/uuid - tests: Update TEST-CACHE-003/004 to document actual behavior (duplicates occur without ObjectIdCache - this is by design) - docs: Correct treatments-schema.md (syncIdentifier/uuid preserved, not copied to identifier) - docs: Remove external link from entries-schema.md Loop carbs/doses rely on ObjectIdCache for dedup, not server-side logic. This matches the original (pre-change) server behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
|
@copilot please review for accuracy. We still see several details that are not accurate including env var behavior? Let's see impact report for these changes as well. |
… docs accuracy Co-authored-by: bewest <394179+bewest@users.noreply.github.com>
fix: gate UUID_HANDLING on both write and read paths; fix doc accuracy
First describe block was missing its closing }); Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
NightscoutKit (Loop) sends profiles wrapped in arrays: [profile]. The MongoDB driver migration changed insert() to insertOne(), breaking array support. Changes: - API layer: normalize input to array, purify each item - Storage layer: use insertMany() instead of insertOne() - Tests: verify single, array, and empty array handling This matches the proven pattern from treatments API. Fixes array handling regression introduced in d46c5b4. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The API layer was calling purifyObject() on the raw req.body without handling arrays. When NightscoutKit sends [status], only the outer array would be purified (no-op), not the individual status objects. Now normalizes to array and purifies each devicestatus object, matching the treatments pattern. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract profile structure from NightscoutKit Swift source: - ProfileSet.swift: dictionaryRepresentation structure - LoopSettings.swift: loopSettings with overrides, dosing params - TemporaryScheduleOverride.swift: override presets format Fixtures include: - minimal: Minimal valid profile - fullLoop: Full profile with loopSettings and schedules - withActiveOverride: Profile with active schedule override - mmol: mmol/L units variant - singleArray: Single profile in array (most common case) - batchArray: 3 profiles for batch upload testing - generateUniqueProfile(): Helper for dedup testing NightscoutKit always sends profiles as arrays (NightscoutClient.swift:404). Sources: externals/NightscoutKit/Sources/NightscoutKit/Models/ProfileSet.swift externals/NightscoutKit/Sources/NightscoutKit/Models/LoopSettings.swift externals/NightscoutKit/Sources/NightscoutKit/Models/TemporaryScheduleOverride.swift Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract devicestatus structure from NightscoutKit Swift source: - DeviceStatus.swift: top-level structure with device, created_at, identifier - LoopStatus.swift: name, version, iob, cob, predicted, enacted, failureReason - PumpStatus.swift: pumpID, manufacturer, model, reservoir, battery - IOBStatus.swift: iob, basaliob with timestamp - COBStatus.swift: cob with timestamp - PredictedBG.swift: startDate, values array, optional COB/IOB curves - LoopEnacted.swift: rate, duration (minutes), received, bolusVolume - UploaderStatus.swift: name, battery percentage - BatteryStatus.swift: percent, voltage, status Fixtures include: - minimal: Just device and timestamp - loopWithIOBCOB: Loop status with IOB/COB - fullLoop: Complete status with pump, predictions, enacted - withEnacted: Status with active temp basal - withFailure: Status with failureReason - withAutoBolus: Status with automatic dose recommendation - withMultiplePredictions: COB and IOB prediction curves - singleArray: Single status in array format - batchArray: 3 statuses for batch upload testing - helpers: Factory functions for building custom fixtures NightscoutKit sends devicestatus as arrays (NightscoutClient.swift:646-647). Sources: externals/NightscoutKit/Sources/NightscoutKit/Models/DeviceStatus.swift externals/NightscoutKit/Sources/NightscoutKit/Models/LoopStatus.swift externals/NightscoutKit/Sources/NightscoutKit/Models/PumpStatus.swift externals/NightscoutKit/Sources/NightscoutKit/Models/IOBStatus.swift externals/NightscoutKit/Sources/NightscoutKit/Models/COBStatus.swift externals/NightscoutKit/Sources/NightscoutKit/Models/PredictedBG.swift externals/NightscoutKit/Sources/NightscoutKit/Models/LoopEnacted.swift Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract treatment fixture patterns from NightscoutKit Swift source: - BolusNightscoutTreatment (Correction Bolus) - CarbCorrectionNightscoutTreatment (Carb Correction) - TempBasalNightscoutTreatment (Temp Basal) - OverrideTreatment (Temporary Override) - MealBolusNightscoutTreatment (Meal Bolus) - BGCheckNightscoutTreatment (BG Check) - NoteNightscoutTreatment (Note) - Site Change, Sensor Start Includes: - Individual treatment objects for each type - Array-wrapped formats (what NightscoutKit actually sends) - Batch upload arrays for testing - Helper functions for generating unique fixtures - syncIdentifier field handling (client-provided dedup ID) Source files: - externals/NightscoutKit/Sources/NightscoutKit/Models/Treatments/*.swift - externals/NightscoutKit/Sources/NightscoutKit/NightscoutClient.swift Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extends api.shape-handling.test.js with: Profile API tests: - Single profile object POST - Single-element array POST (NightscoutKit format) - Multi-element array POST (batch upload) - Response count equals input count validation - Empty array handling - Response shape consistency (always returns array) NightscoutKit Fixtures Integration tests: - Bolus treatments with syncIdentifier - Carb entries with absorption time - Temp basal treatments - Mixed batch arrays - Loop devicestatus with IOB/COB - Batch devicestatus arrays - Loop profile with loopSettings - Historical profile batch sync Test matrix coverage per spec in profile-api-array-regression.md: | API | Single | Array | Batch | Empty | Response=Array | _id present | |-----|--------|-------|-------|-------|----------------|-------------| | Treatments | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | DeviceStatus | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Entries | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Profile | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add validation for _id field in profile API: - POST: validates each document's _id before storage - PUT: validates _id format before update - DELETE: validates _id parameter before removal Accepts: undefined, null, or 24-character hex string Rejects: UUIDs, short strings, numbers, objects with 400 Bad Request This prevents 500 errors from BSONError when clients send UUID-style _ids (e.g., NightscoutKit). Tests added for all validation cases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add validation for _id field in devicestatus API: - POST: validates each document's _id before storage - DELETE: validates _id parameter (allows wildcard '*') Accepts: undefined, null, or 24-character hex string Rejects: UUIDs, short strings, numbers, objects with 400 Bad Request Previously, invalid _id values were silently stored as strings instead of ObjectIds, causing inconsistent data and query issues. Tests added for all validation cases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add validation for _id field in activity and food APIs: - activity: POST, PUT, DELETE now validate _id format - food: POST, PUT, DELETE now validate _id format Accepts: undefined, null, or 24-character hex string Rejects: UUIDs, short strings, numbers, objects with 400 Bad Request Previously: - activity: 500 crash on invalid _id in save/remove - food: silently replaced invalid _id with new ObjectId (data loss) Tests added covering all validation cases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add array normalization to food API POST endpoint:
- API layer: normalize single object to array (like activity/profile)
- Storage layer: use replaceOne loop with upsert (same as activity pattern)
- Storage layer: accept both single object and array for backward compat
Previously POST /api/food/ with array input would crash:
insertOne([{...}]) → MongoDB error
Now supports both single object and array input consistently.
Response format is now array (matching treatments pattern).
Fixes #8447
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace sequential replaceOne calls with bulkWrite for batch upserts. This improves performance when inserting multiple documents at once. - activity.js: Use bulkWrite with replaceOne ops instead of forEach loop - food.js: Same optimization for consistency Both maintain exact same upsert behavior (query by _id+created_at or doc). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
|
@copilot Can we verify in a |
- Add tests for POST with array of foods - Add tests for PUT with array of foods - Add test for empty array returning empty array - Fix PUT endpoint to normalize array input like POST - Fix food.save() storage to handle arrays with bulkWrite - Rename test file to follow *.test.js convention Validates fix from ef7bff3 for complete array handling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add test for single activity returns array with one item - Add test for activity array returns array - Add test for empty array returns empty array - Rename test file to follow *.test.js convention Validates array normalization behavior for activity API. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add test for single entry returns array with one item - Add test for empty array returns empty result - Validates response format consistency for entries API Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Additional tweaks to try to fully prep for release:
UUID_HANDLINGfor both reads and writes.