Skip to content

Wip/bewest/mongodb 5x#8421

Merged
AndyLow91 merged 169 commits intonightscout:devfrom
bewest:wip/bewest/mongodb-5x
Mar 16, 2026
Merged

Wip/bewest/mongodb 5x#8421
AndyLow91 merged 169 commits intonightscout:devfrom
bewest:wip/bewest/mongodb-5x

Conversation

@bewest
Copy link
Member

@bewest bewest commented Jan 19, 2026

Updates iterating on the mongodb driver.

bewest and others added 30 commits January 19, 2026 13:09
Update project documentation and startup script for Nightscout v15.0.4, including Replit specific configurations like INSECURE_USE_HTTP=true and webpack bundling.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 748e831e-92ae-4927-9569-47665f47f29c
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 6e1d3764-af75-4a2b-a75f-f7252955326d
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/7ea4278b-5c6c-4065-9cb8-f1013771318d/748e831e-92ae-4927-9569-47665f47f29c/FO8cDm7
Replit-Helium-Checkpoint-Created: true
Enhance `replit.md` to include comprehensive documentation on API endpoints (v1, v2, v3), authentication methods (API_SECRET, JWT), OpenAPI specs, Socket.IO channels, and environment variables.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 748e831e-92ae-4927-9569-47665f47f29c
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 759a1536-7b1b-4ce2-9972-168703ba4f7d
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/7ea4278b-5c6c-4065-9cb8-f1013771318d/748e831e-92ae-4927-9569-47665f47f29c/FO8cDm7
Replit-Helium-Checkpoint-Created: true
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 06715f81-f852-4156-8ab0-cf9e0aa3564d
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 79dc8047-74ae-47e7-85a6-8c7255d9603d
Create a new RFC document and associated JSON schemas defining an agentic control plane for automated insulin delivery systems, including event envelopes, capabilities models, delivery tracking, and conflict rules.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 06715f81-f852-4156-8ab0-cf9e0aa3564d
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: 010cdc4a-2c86-4d2e-9768-bf2769a89953
Replit-Helium-Checkpoint-Created: true
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 06715f81-f852-4156-8ab0-cf9e0aa3564d
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: a40997df-4db2-4767-9c35-abb942a5fca2
Replit-Helium-Checkpoint-Created: true
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 0dc64f83-dd07-4193-899d-2998f5eb61c8
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 31a7d26d-5532-4bb7-90f2-0c00bf18017e
Replit-Helium-Checkpoint-Created: true
Create a proposal document detailing the current state of the test suite, dependency analysis, and a phased strategy for modernization, including migrating client tests to Jest.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 0dc64f83-dd07-4193-899d-2998f5eb61c8
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: 560c5b85-bd12-4a58-bc27-d8f6612621a6
Replit-Helium-Checkpoint-Created: true
Update replit.md to include a new proposal for modernizing test infrastructure, detailing a three-phase strategy and critical dependency updates.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 0dc64f83-dd07-4193-899d-2998f5eb61c8
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 3fb6e201-20e1-4478-9e20-72faa517ec04
Replit-Helium-Checkpoint-Created: true
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 6bd7e260-440b-4cf7-bd4f-6c49f415222f
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 003afa1a-8cd2-4c59-81d5-c8169366a6cf
Replit-Helium-Checkpoint-Created: true
…ral goals

Revise the testing modernization proposal to include interview findings, a new three-track approach, and scope control guardrails.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 6bd7e260-440b-4cf7-bd4f-6c49f415222f
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: d3a329ea-f588-48b4-ab9f-72250d9f0d39
Replit-Helium-Checkpoint-Created: true
Revise testing modernization proposal to include UI modernization, architecture goals, and a three-track approach.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 6bd7e260-440b-4cf7-bd4f-6c49f415222f
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 6175529f-d563-44d9-a0d8-39ef3f387bd8
Replit-Helium-Checkpoint-Created: true
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 6faed36e-345f-45eb-8006-45dc67e8451e
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 8124194f-045a-4644-b523-2697d2284678
Replit-Helium-Checkpoint-Created: true
…ity measures

Corrects inaccuracies in the Security Audit, Architecture Overview, and Modernization Roadmap documentation regarding rate limiting, Node.js versions, and code examples.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 6faed36e-345f-45eb-8006-45dc67e8451e
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: 207079d8-1a78-4ade-ad5c-d33397f8c775
Replit-Helium-Checkpoint-Created: true
Add a comprehensive system audit documentation section to replit.md, detailing findings on rate limiting, dependencies, bundle size, and Node.js support, along with a list of 9 audit documents.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 6faed36e-345f-45eb-8006-45dc67e8451e
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: a4473413-4cff-4847-b61c-6fc7e4d66b0a
Replit-Helium-Checkpoint-Created: true
Modify docs/modernization-roadmap.md to replace time intervals with effort (low/high) and complexity (straightforward/complicated) descriptions for each phase and task.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 6faed36e-345f-45eb-8006-45dc67e8451e
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 8aa5ae46-3944-4b57-9c5f-7f7df957037e
Replit-Helium-Checkpoint-Created: true
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 6faed36e-345f-45eb-8006-45dc67e8451e
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 02e98f82-c851-43a4-b518-90eccf9df206
Replit-Helium-Checkpoint-Created: true
Update architecture, security, and modernization documentation to clarify existing brute-force protection, introduce OIDC/OAuth2 integration plans, and address rate limiting gaps.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 6faed36e-345f-45eb-8006-45dc67e8451e
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: 483af0ac-57e5-44ad-89c0-d2374643791a
Replit-Helium-Checkpoint-Created: true
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 6faed36e-345f-45eb-8006-45dc67e8451e
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 0a9ae46c-4b3a-4953-b56d-f33d485ea783
Replit-Helium-Checkpoint-Created: true
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: d9c34e72-a841-40c0-9e5f-c5e839501957
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 151d4417-e878-49d8-8040-5dcaf8a7da08
Replit-Helium-Checkpoint-Created: true
Add new sections for Pagination Metadata, Date Format Normalization, and Observability Plan to the API query normalization proposal document, and renumber existing sections accordingly.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: d9c34e72-a841-40c0-9e5f-c5e839501957
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: 9d148101-339f-436a-bbbe-56da2f911abb
Replit-Helium-Checkpoint-Created: true
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: d9c34e72-a841-40c0-9e5f-c5e839501957
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 147a1f33-7ada-4bb0-b6c6-9c89c8c0f7dc
Replit-Helium-Checkpoint-Created: true
Replace timeline-based phases with complexity and risk ratings in the proposal document.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: d9c34e72-a841-40c0-9e5f-c5e839501957
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: aef39c0d-7cd4-4b3c-87f7-303fcc84a898
Replit-Helium-Checkpoint-Created: true
…ss_frequent_db_updates""

This reverts commit 2e65b1d.
Try to bring back nightscout#8026.
Provide some tests for the profile API.
Refactor `devicestatus.js` to use `async.eachSeries` for sequential processing and update WebSocket `dbAdd` handler to process array inputs sequentially, ensuring correct multi-document write support.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: fbbf36df-818c-4b8c-8760-4975515f38e6
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 522116f1-cf99-4504-9c93-bc47b4276585
Replit-Helium-Checkpoint-Created: true
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: fbbf36df-818c-4b8c-8760-4975515f38e6
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 8a0072a5-b8d3-4d98-977e-5f56d3d1d195
Replit-Helium-Checkpoint-Created: true
Copilot AI added a commit that referenced this pull request Mar 8, 2026
Co-authored-by: bewest <394179+bewest@users.noreply.github.com>
…nsparent promotion

Fixes nightscout#8450 - Loop Temporary Override sync breaks due to UUID _id handling

Option G Implementation:
- Extract client sync identity (identifier) from any source:
  - Loop overrides: UUID in _id field → moved to identifier
  - Loop carbs/doses: syncIdentifier → copied to identifier
  - AAPS: identifier already present
  - xDrip+: uuid → copied to identifier

- Server generates proper ObjectId for _id field
- Deduplication uses identifier (not _id) as primary key
- No database migration needed - gradual adoption

Changes:
- normalizeTreatmentId(): extracts client identity to identifier field
- upsertQueryFor(): identifier-first lookup, strips UUID _id for upsert
- create()/upsert()/save(): fetch _id from DB after update by identifier
- Added 'identifier' to indexedFields for efficient querying
- Updated UUID treatment test with full workflow coverage

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
sergev-art pushed a commit to sergev-art/cgm-remote-monitor that referenced this pull request Mar 10, 2026
Co-authored-by: bewest <394179+bewest@users.noreply.github.com>
bewest and others added 7 commits March 10, 2026 14:54
TEST-GAP-001: Loop override POST with UUID _id
TEST-GAP-002: Loop override DELETE by UUID
TEST-GAP-003: Loop override UPDATE by UUID
TEST-GAP-004: Loop override re-POST (upsert)

12 new tests validating REQ-SYNC-072 behavior:
- UUID _id promoted to identifier field
- Server generates valid ObjectId for _id
- Updates/deletes work via identifier lookup
- Duplicate detection via identifier
- Batch and edge case handling

New fixtures:
- loop-override.js: Real Loop override payload patterns

All 12 tests passing.

Refs: GAP-TREAT-012, REQ-SYNC-072

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TEST-CACHE-001: POST carb → cache syncIdentifier → PUT with id
TEST-CACHE-002: POST dose → cache syncIdentifier → DELETE with id
TEST-CACHE-003: Cache miss (24hr expiry) → POST same syncIdentifier
TEST-CACHE-004: App restart (cache empty) → POST existing syncIdentifier
TEST-CACHE-005: Batch POST → verify response order → cache mapping

7 new tests validating Loop's ObjectIdCache behavior:
- syncIdentifier → ObjectId mapping
- Response order for batch operations
- Deduplication by syncIdentifier
- Hex string syncIdentifier handling

All 7 tests passing.

Refs: Loop ObjectIdCache.swift analysis

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Client Pattern Tests:
- TEST-ID-001: Loop Override UUID _id → identifier
- TEST-ID-002: Loop Override identifier field
- TEST-ID-003: Loop Carb syncIdentifier
- TEST-ID-004: AAPS identifier: null
- TEST-ID-005: AAPS identifier: ObjectId
- TEST-ID-006: xDrip+ uuid + _id fields

v1 API Identity Tests:
- TEST-V1-ID-001: No id field
- TEST-V1-ID-002: Valid ObjectId
- TEST-V1-ID-003: UUID string (REQ-SYNC-072)
- TEST-V1-ID-004: syncIdentifier field

Deduplication Tests:
- Duplicate identifier handling
- Unique identifiers create separate docs

12 new tests, all passing.

Refs: REQ-SYNC-072, GAP-TREAT-012

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Carb Tests (TEST-CARB-001 to 004):
- Create carb with syncIdentifier and absorptionTime
- Create carb with fat/protein (Warsaw FPU method)
- Create carb with cached _id
- Update carb via cached _id
- Delete carb via cached _id

Dose Tests (TEST-DOSE-001 to 005):
- Bolus with syncIdentifier
- Meal bolus with carbs and insulin
- Temp basal with rate and duration
- Suspend (zero rate) temp basal
- Update dose via cached _id
- Hex string syncIdentifier (pump events)
- Mixed dose batch maintains order

13 new tests, all passing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
SGV Entry Tests (TEST-SGV-001 to 005):
- Single SGV with required fields
- All direction values
- Loop device identifier preserved
- Dexcom device with filtered/unfiltered/noise
- SGV deduplication by date+device
- Different devices create separate entries
- MBG (manual BG check) entries

DeviceStatus Tests (TEST-DS-001 to 005):
- Loop status with IOB/COB
- Loop predicted values array
- Loop enacted temp basal
- Pump reservoir and battery
- Omnipod specific fields
- Override in deviceStatus
- Override with insulinNeedsScaleFactor
- Complete Loop deviceStatus

17 new tests, all passing (716 total).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add 3 new tests for explicit identifier field handling:
- supports identifier field for AAPS-style treatments
- deduplicates by identifier on re-upload
- supports batch upload with identifiers

These complement existing UUID _id tests (Loop pattern) to cover
both AID client sync patterns.

Refs: REQ-SYNC-072, GAP-TREAT-012

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TEST-V3-ID-001: Null identifier generates ObjectId, copies to identifier
TEST-V3-ID-002: ObjectId string as identifier uses it directly
TEST-V3-ID-003: UUID string identifier preserved as-is

Validates Option G behavior extends to v3 API endpoints.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
bewest and others added 2 commits March 10, 2026 16:35
Trio/Loop upload CGM entries with UUID strings as _id field.
This caused MongoDB errors when re-uploading with different UUID
at same timestamp: "immutable field '_id'" error.

Fix:
- Add normalizeEntryId() to extract UUID from _id to identifier field
- Add upsertQueryFor() to strip non-ObjectId _id before $set
- Maintain sysTime+type as primary dedup key for CGM data integrity
- Add identifier to indexed fields

Tests:
- 3 baseline tests document current sysTime+type dedup behavior
- 6 UUID handling tests including the previously-failing scenario

Refs: GAP-SYNC-045, REQ-SYNC-072

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@bewest
Copy link
Member Author

bewest commented Mar 11, 2026

Test Growth Summary:

Metric Value
Before MongoDB work (Jan 2026) 486 tests
Current (pr-8447) 731 tests
Tests added +245
Growth +50.4%
New test files 21

New Test Files Added:

Category Files Focus
AID Client Tests api.aaps-client.test.js, api3.aaps-patterns.test.js, identity-matrix.test.js Client compatibility
UUID/Sync Tests api.entries.uuid.test.js, gap-treat-012.test.js, objectid-cache.test.js GAP-SYNC-045, GAP-TREAT-012
Dedup/Batch Tests api.deduplication.test.js, api.v1-batch-operations.test.js, concurrent-writes.test.js MongoDB modernization
Shape Handling api.shape-handling.test.js, api3.shape-handling.test.js, storage.shape-handling.test.js, websocket.shape-handling.test.js Data integrity
Loop/Trio Tests carb-dose-upload.test.js, sgv-devicestatus.test.js iOS app uploads
Infrastructure api.profiles.test.js, mongo-pool-config.test.js, flakiness-control.test.js Stability

bewest added a commit to bewest/rag-nightscout-ecosystem-alignment that referenced this pull request Mar 12, 2026
Creates work tracking for reviewer's guide completion:
- 26 library files categorized
- 10 test file groups
- 6 documentation audit items
- 20-iteration plan with ~30 min per iteration

Links to: docs/PR-8421-reviewers-guide.md
Worktree: /home/bewest/src/worktrees/nightscout/cgm-pr-8447
PR: nightscout/cgm-remote-monitor#8421

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
bewest and others added 3 commits March 12, 2026 12:24
SAFETY-001: Fix tests/ci.test.env to use NODE_ENV=test instead of production
SAFETY-002: Add NODE_ENV check to tests/hooks.js with warning
SAFETY-003: Create tests/fixtures/test-guard.js with guarded deleteMany/drop helpers

This prevents deleteMany({}) from accidentally running against production
databases if test environment is misconfigured.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Change from warning to process.exit(1) to prevent any possibility of
running destructive test operations against a production database.

Tests now fail immediately if NODE_ENV !== 'test', with clear instructions
on how to fix.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
GitHub deprecated Node 20 on runners (2025-09-19), but branch protection
rules still require Node 20 tests to pass. Set ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true
only for the test job, not publish.

Reverts node-version matrix to [20, 22] to match master.

Ref: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@bewest
Copy link
Member Author

bewest commented Mar 12, 2026

https://github.com/bewest/rag-nightscout-ecosystem-alignment/blob/main/docs/PR-8421-reviewers-guide.md

I created some notes to help anyone reviewing this patch set. The majority of changes are docs and tests. The docs could be squashed or rebased onto another pr but I believe this would just be more work without any benefit. The core changes are ~1k lines of changes to accommodate backwards compatibility while enabling mongo upgrade without any ill effects. The review guide may help with this.

bewest and others added 2 commits March 12, 2026 14:09
The 'WebSocket dbAdd Array Handling Investigation' block was R&D to
understand insertOne behavior with arrays. The investigation concluded:
- MongoDB's insertOne([a,b]) creates single doc (not multiple)
- Fix: sequential processing via processNextItem() in websocket.js

Production tests now cover this behavior:
- 'dbAdd with array input for treatments - current behavior test'
- 'dbAdd with array input for devicestatus - current behavior test'
- 'dbAdd with array input for entries - current behavior test'

Removes 2 flaky investigative tests, keeps 729 production tests passing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This test passes locally in ~50ms but occasionally times out at 30s
in constrained GitHub runners. Adding retries(2) allows it to recover
from transient CI resource contention.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@AndyLow91 AndyLow91 merged commit cdef1e7 into nightscout:dev Mar 16, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants