Skip to content

Commit 7e5c345

Browse files
bewestbewest
authored
Merge pull request #8456 from nightscout/wip/test-improvements
Wip/test improvements - extends same fixes for data shape across all remaining api surface areas and includes test coverage across the test matrix spectrum.
2 parents cdef1e7 + 729a6f5 commit 7e5c345

34 files changed

+3483
-162
lines changed

‎CHANGELOG.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Changelog
2+
3+
All notable changes to cgm-remote-monitor are documented in this file.
4+
5+
## [15.0.7] - 2026-03-XX (Unreleased)
6+
7+
### Added
8+
9+
#### UUID/Identifier Handling (REQ-SYNC-072)
10+
11+
- **`UUID_HANDLING` env var** (default: `true`): Feature flag that controls UUID `_id` normalization for treatments and entries.
12+
- When `true`: UUID values sent as `_id` are extracted to the `identifier` field and a server-generated ObjectId is assigned. GET/DELETE by UUID are routed through the `identifier` field.
13+
- When `false`: UUID `_id` values are stripped (UUID identity not preserved) and UUID-based queries return empty results.
14+
- **Treatments API**: Loop overrides with UUID `_id` are now normalized correctly, preventing duplicate records (Issue #8450).
15+
- **Entries API**: CGM entries (e.g., Trio) with UUID `_id` are now handled correctly.
16+
- **Scope**: Only UUID values in the `_id` field are affected. Other client identity fields (`syncIdentifier`, `uuid`, `identifier`) are preserved but not modified.
17+
18+
#### Test Infrastructure
19+
20+
- **NODE_ENV=test safety check**: Tests now refuse to run without `NODE_ENV=test`, preventing accidental production database modification.
21+
- Comprehensive test suite for UUID handling behavior across write and read paths.
22+
23+
### Documentation
24+
25+
- Updated README.md with `UUID_HANDLING` and MongoDB pool configuration env vars.
26+
- Added entries schema documentation (`docs/data-schemas/entries-schema.md`).
27+
- Updated treatments schema documentation with identifier normalization behavior.
28+
- Added test environment variables reference to CONTRIBUTING.md.
29+
30+
---
31+
32+
## [15.0.6] - Previous Release
33+
34+
See GitHub releases for prior changelog entries.

‎CONTRIBUTING.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,69 @@ Once `dev` has been reviewed and people feel it's time to release, we follow the
142142

143143
Every commit is tested by travis. We encourage adding tests to validate your design. We encourage discussing your use cases to help everyone get a better understanding of your design.
144144

145+
## Running Tests Locally
146+
147+
Tests **require** `NODE_ENV=test` to protect production databases from accidental destruction. If this variable is not set, tests will refuse to run and exit with an error.
148+
149+
```bash
150+
# Run all tests
151+
NODE_ENV=test npm test
152+
153+
# Run only unit tests (parallel, fast)
154+
NODE_ENV=test npm run test:unit
155+
156+
# Run only integration tests (sequential, needs MongoDB)
157+
NODE_ENV=test npm run test:integration
158+
159+
# Run specific test files
160+
NODE_ENV=test npm test -- --grep "treatments"
161+
```
162+
163+
### Advanced Test Scripts
164+
165+
For diagnosing test issues and ensuring reliability:
166+
167+
```bash
168+
# Run stress tests for concurrent write operations
169+
NODE_ENV=test npm run test:stress
170+
171+
# Detect flaky tests by running multiple iterations
172+
NODE_ENV=test npm run test:flaky # Default iterations
173+
NODE_ENV=test npm run test:flaky:quick # 3 iterations
174+
NODE_ENV=test npm run test:flaky:thorough # 20 iterations
175+
176+
# Run specific flaky test harnesses
177+
NODE_ENV=test npm run test:flaky:entries # Entries isolation tests
178+
NODE_ENV=test npm run test:flaky:socket # Socket isolation tests
179+
180+
# Enable timing warnings to find slow tests
181+
NODE_ENV=test npm run test:timing
182+
```
183+
184+
You can create a `my.test.env` file based on `ci.test.env` for local testing:
185+
186+
```bash
187+
source my.test.env && npm test
188+
```
189+
190+
### Test Environment Variables
191+
192+
These variables control test behavior and MongoDB connection pooling:
193+
194+
| Variable | Purpose | Default |
195+
|----------|---------|---------|
196+
| `NODE_ENV=test` | **Required** - Enables test mode, prevents production DB access | - |
197+
| `MONGO_POOL_SIZE` | MongoDB connection pool size | 5 |
198+
| `MONGO_MIN_POOL_SIZE` | Minimum pool connections to keep open | 0 |
199+
| `MONGO_MAX_IDLE_TIME_MS` | Max idle time (ms) before closing connection | 30000 |
200+
| `AUTH_FAIL_DELAY` | Delay (ms) after auth failure (test speedup) | 5000 |
201+
202+
For CI or resource-constrained environments, adjust pool size:
203+
204+
```bash
205+
MONGO_POOL_SIZE=3 MONGO_MIN_POOL_SIZE=1 npm test
206+
```
207+
145208
## Other Dev Tips
146209

147210
* Join the [Discord chat][discord-url].

‎README.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -161,8 +161,8 @@ Older versions or other browsers might work, but are untested and unsupported. W
161161

162162
## Installation software requirements:
163163

164-
- [Node.js](http://nodejs.org/) Latest Node v14 or v16 LTS. Node versions that do not have the latest security patches will not be supported. Use [Install instructions for Node](https://nodejs.org/en/download/package-manager/) or use `bin/setup.sh`)
165-
- [MongoDB](https://www.mongodb.com/download-center?jmp=nav#community) 4.2 or 4.4.
164+
- [Node.js](http://nodejs.org/) Node v20 LTS or later (v22, v24 also supported). Node versions that do not have the latest security patches will not be supported. Use [Install instructions for Node](https://nodejs.org/en/download/package-manager/) or use `bin/setup.sh`)
165+
- [MongoDB](https://www.mongodb.com/download-center?jmp=nav#community) 4.4 or later (5.0, 6.0 also supported).
166166

167167
As a non-root user clone this repo then install dependencies into the root of the project:
168168

@@ -252,6 +252,7 @@ To learn more about the Nightscout API, visit https://YOUR-SITE.com/api-docs/ or
252252
Setting it to `denied` will require a token from every visit, using `status-only` will enable api-secret based login.
253253
* `IMPORT_CONFIG` - Used to import settings and extended settings from a url such as a gist. Structure of file should be something like: `{"settings": {"theme": "colors"}, "extendedSettings": {"upbat": {"enableAlerts": true}}}`
254254
* `TREATMENTS_AUTH` (`on`) - possible values `on` or `off`. Deprecated, if set to `off` the `careportal` role will be added to `AUTH_DEFAULT_ROLES`
255+
* `UUID_HANDLING` (`true`) - Controls how UUID `_id` values are handled for treatments and entries. When `true` (default), if a client sends a UUID string as the `_id` field, it is extracted to the `identifier` field (for sync deduplication) and the server generates a proper ObjectId for `_id`. Queries by UUID (`GET`/`DELETE`) are also routed through the `identifier` field. When `false`, UUID `_id` values are silently stripped on write (no identifier is preserved) and UUID-based queries return empty results. This only affects the specific case where a UUID is sent as `_id` (e.g., Loop overrides, Trio CGM entries).
255256

256257
#### Data Rights
257258

@@ -288,6 +289,9 @@ autonomy for your data:
288289
* `MONGO_PROFILE_COLLECTION`(`profile`) - The collection used to store your profiles
289290
* `MONGO_FOOD_COLLECTION`(`food`) - The collection used to store your food database
290291
* `MONGO_ACTIVITY_COLLECTION`(`activity`) - The collection used to store activity data
292+
* `MONGO_POOL_SIZE` (`5`) - MongoDB connection pool size. Adjust for your deployment needs.
293+
* `MONGO_MIN_POOL_SIZE` (`0`) - Minimum pool connections to keep open.
294+
* `MONGO_MAX_IDLE_TIME_MS` (`30000`) - Max idle time (ms) before closing a connection.
291295
* `PORT` (`1337`) - The port that the node.js application will listen on.
292296
* `HOSTNAME` - The hostname that the node.js application will listen on, null by default for any hostname for IPv6 you may need to use `::`.
293297
* `SSL_KEY` - Path to your ssl key file, so that ssl(https) can be enabled directly in node.js. If using Let's Encrypt, make this variable the path to your privkey.pem file (private key).

‎docs/data-schemas/entries-schema.md‎

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
# Entries Schema Documentation
2+
3+
**Document Version:** 1.0
4+
**Last Updated:** March 2026
5+
**Status:** Active (2025 Standard)
6+
**Source:** Code analysis (`lib/server/entries.js`)
7+
8+
---
9+
10+
## Overview
11+
12+
The `entries` collection stores CGM (Continuous Glucose Monitor) sensor readings and related data. This includes SGV (sensor glucose values), MBG (meter blood glucose), and calibration data.
13+
14+
**Collection Name:** `entries`
15+
**Primary Key:** `sysTime` + `type` (composite)
16+
**Primary Timestamp Field:** `sysTime` (ISO 8601, derived from `dateString` or `date`)
17+
18+
---
19+
20+
## Core Fields
21+
22+
| Field | Type | Required | Constraints | Description |
23+
|-------|------|----------|-------------|-------------|
24+
| `_id` | ObjectId | Yes (auto) | MongoDB ObjectId | Primary key, auto-generated by server |
25+
| `type` | String | Yes | `sgv`, `mbg`, `cal`, etc. | Type of entry |
26+
| `date` | Number | Yes | Epoch milliseconds | When the reading was taken |
27+
| `dateString` | String | Yes | ISO 8601 | Same as `date` in string format |
28+
| `sysTime` | String | Computed | ISO 8601 | Normalized timestamp (computed from `dateString` or `date`) |
29+
| `utcOffset` | Number | Computed | Minutes | UTC offset parsed from `dateString` |
30+
31+
---
32+
33+
## SGV Fields (type: "sgv")
34+
35+
| Field | Type | Constraints | Description |
36+
|-------|------|-------------|-------------|
37+
| `sgv` | Number | mg/dL or mmol/L | Sensor glucose value |
38+
| `direction` | String | Trend arrows | Glucose trend direction |
39+
| `noise` | Number | 0-4 | Signal noise level |
40+
| `filtered` | Number | Raw value | Filtered sensor signal |
41+
| `unfiltered` | Number | Raw value | Unfiltered sensor signal |
42+
| `rssi` | Number | dBm | Signal strength (Dexcom) |
43+
44+
---
45+
46+
## Sync Identity Fields
47+
48+
| Field | Type | Source | Description |
49+
|-------|------|--------|-------------|
50+
| `identifier` | String | Server-normalized | **Unified client sync identity** (see below) |
51+
| `device` | String | CGM app | Device/app identifier (e.g., `"xDrip-DexcomG6"`) |
52+
53+
### Identifier Field Normalization (REQ-SYNC-072)
54+
55+
As of v15.0.7, the server normalizes UUID values in `_id` into the `identifier` field when `UUID_HANDLING=true` (default):
56+
57+
| Client | Sends | Server Action (UUID_HANDLING=true) | Server Action (UUID_HANDLING=false) |
58+
|--------|-------|-------------------------------------|--------------------------------------|
59+
| **Trio** | UUID in `_id` | Move to `identifier`, assign server ObjectId | Strip `_id`, assign ObjectId (UUID not preserved) |
60+
| **Loop** (entries) | ObjectId (from cache) | Normal ObjectId behavior | Normal ObjectId behavior |
61+
62+
**Note**: The `UUID_HANDLING` env var controls **both** write-path normalization (identifier extraction) and read-path queries (GET/DELETE by UUID).
63+
64+
**Important**: For entries, `sysTime + type` is ALWAYS the primary deduplication key. The `identifier` field is for client sync tracking only - it does NOT override the dedup logic.
65+
66+
---
67+
68+
## Deduplication Behavior
69+
70+
Entries use **sysTime + type** as the composite unique key:
71+
72+
```javascript
73+
// Upsert query for entries (lib/server/entries.js)
74+
{ sysTime: doc.sysTime, type: doc.type }
75+
```
76+
77+
This means:
78+
- Two SGV readings at the same `sysTime` will be deduplicated (one overwrites the other)
79+
- Different entry types (SGV vs MBG) at the same time are allowed
80+
- Re-uploading the same reading updates the existing document
81+
82+
### UUID _id Handling
83+
84+
When a client sends a UUID as `_id`:
85+
86+
1. **Extract**: UUID is copied to `identifier` field (when `UUID_HANDLING=true`)
87+
2. **Strip**: Non-ObjectId `_id` is removed before database operation
88+
3. **Upsert**: Server uses `sysTime + type` for matching
89+
4. **Assign**: Server-generated ObjectId becomes final `_id`
90+
91+
This prevents the MongoDB "immutable field '_id'" error while preserving client sync identity (when enabled).
92+
93+
---
94+
95+
## UUID_HANDLING Feature Flag
96+
97+
The `UUID_HANDLING` environment variable controls both **write-path** normalization (identifier extraction) and **read-path** queries (GET/DELETE by UUID).
98+
99+
When `UUID_HANDLING=true` (default):
100+
101+
| Operation | Behavior |
102+
|-----------|----------|
103+
| POST/PUT with UUID `_id` | UUID moved to `identifier`, server assigns ObjectId |
104+
| GET by UUID | Searches by `identifier` field |
105+
| DELETE by UUID | Deletes by `identifier` field |
106+
107+
When `UUID_HANDLING=false`:
108+
109+
| Operation | Behavior |
110+
|-----------|----------|
111+
| POST/PUT with UUID `_id` | UUID `_id` stripped, ObjectId assigned (UUID not preserved) |
112+
| GET by UUID | Returns empty (no crash) |
113+
| DELETE by UUID | Deletes nothing (no crash) |
114+
115+
**Note**: This only affects cases where a UUID is passed as the `_id` field (writes) or as the `_id` parameter in API calls (reads), e.g., `GET /api/v1/entries/{uuid}`.
116+
117+
---
118+
119+
## Example Entry Document
120+
121+
```json
122+
{
123+
"_id": "507f1f77bcf86cd799439011",
124+
"type": "sgv",
125+
"sgv": 120,
126+
"direction": "Flat",
127+
"date": 1704067200000,
128+
"dateString": "2024-01-01T00:00:00.000Z",
129+
"sysTime": "2024-01-01T00:00:00.000Z",
130+
"utcOffset": 0,
131+
"identifier": "550e8400-e29b-41d4-a716-446655440000",
132+
"device": "xDrip-DexcomG6",
133+
"noise": 1
134+
}
135+
```
136+
137+
---
138+
139+
## Related Documents
140+
141+
- [Treatments Schema](treatments-schema.md) - Similar identifier normalization
142+
- [GAP-SYNC-045](../../../traceability/sync-identity-gaps.md#gap-sync-045) - Trio entries UUID issue
143+
- [REQ-SYNC-072](../../../traceability/sync-identity-requirements.md#req-sync-072) - Identifier normalization requirement

‎docs/data-schemas/treatments-schema.md‎

Lines changed: 31 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -114,18 +114,41 @@ The `treatments` collection stores all user interventions and system events rela
114114
|-------|------|--------|-------------|
115115
| `srvCreated` | String (ISO 8601) | Server | When the server first received this record |
116116
| `srvModified` | String (ISO 8601) | Server | When the server last modified this record |
117-
| `identifier` | String | AAPS | AAPS-specific unique identifier for sync |
118-
| `uuid` | String | Various | Client-assigned unique identifier |
117+
| `identifier` | String | Server/Client | Unified sync identity (see below) |
118+
| `syncIdentifier` | String | Loop | Loop carbs/doses sync identity (preserved, not copied) |
119+
| `uuid` | String | xDrip+ | xDrip+ sync identity (preserved, not copied) |
119120
| `pumpId` | String | Loop/pumps | Pump-assigned identifier |
120121
| `pumpType` | String | Loop/pumps | Type of pump that created this treatment |
121122
| `pumpSerial` | String | Loop/pumps | Serial number of the pump |
122123

123-
**Important:** Different controller systems use different field names for duplicate detection:
124-
- **AAPS:** Uses `identifier` field
125-
- **Loop:** Uses `_id` or pump-related fields
126-
- **xDrip:** Uses `uuid` field
124+
### Identifier Field Normalization (REQ-SYNC-072)
127125

128-
This inconsistency suggests that **schema registration / inversion of control** would be valuable - controllers should register their sync identity field conventions.
126+
As of v15.0.7, the server normalizes **UUID values in the `_id` field** when `UUID_HANDLING=true` (default):
127+
128+
| Client | Client Field | UUID_HANDLING=true | UUID_HANDLING=false |
129+
|--------|--------------|---------------------|----------------------|
130+
| **Loop** (overrides) | UUID in `_id` | Move to `identifier`, assign ObjectId | Strip `_id`, assign ObjectId (UUID not preserved) |
131+
| **Loop** (carbs/doses) | `syncIdentifier` | Preserved as-is | Preserved as-is |
132+
| **AAPS** | `identifier` | Unchanged (already correct) | Unchanged (already correct) |
133+
| **xDrip+** | `uuid` | Preserved as-is | Preserved as-is |
134+
135+
**Scope:** Only UUID values in the `_id` field are affected. Other client identity fields (`syncIdentifier`, `uuid`) are preserved but NOT copied to `identifier`.
136+
137+
**UUID_HANDLING controls both write-path normalization and read-path queries** (GET/DELETE by UUID `_id`).
138+
139+
**Deduplication Priority:** The server uses `identifier` or `_id` for upsert matching when present, falling back to `created_at + eventType` for legacy records.
140+
141+
**Example - Loop Override Upload (UUID_HANDLING=true):**
142+
143+
```javascript
144+
// Client sends:
145+
{ "_id": "A1B2C3D4-E5F6-7890-ABCD-EF1234567890", "eventType": "Temporary Override", ... }
146+
147+
// Server stores:
148+
{ "_id": ObjectId("..."), "identifier": "A1B2C3D4-E5F6-7890-ABCD-EF1234567890", "eventType": "Temporary Override", ... }
149+
```
150+
151+
**Note:** Loop carbs/doses (which use `syncIdentifier`) rely on Loop's local ObjectIdCache for dedup, not server-side logic.
129152

130153
---
131154

@@ -349,4 +372,5 @@ The defaults documented (eventType defaulting to `<none>`, created_at defaulting
349372

350373
| Date | Author | Changes |
351374
|------|--------|---------|
375+
| 2026-03-17 | Agent | Added identifier field normalization (REQ-SYNC-072) |
352376
| 2026-01-15 | Agent | Initial schema documentation from code analysis and domain expert interview |

‎docs/example-template.env‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,4 +12,17 @@ LANGUAGE=en
1212
INSECURE_USE_HTTP=true
1313
PORT=1337
1414
NODE_ENV=development
15-
AUTH_FAIL_DELAY=50
15+
AUTH_FAIL_DELAY=50
16+
17+
# UUID handling for specific client patterns that send UUID as _id field
18+
# Only affects cases where a UUID is sent as the _id field itself
19+
# (e.g., Loop overrides, Trio CGM entries)
20+
# Does NOT affect clients using separate identifier/uuid fields (AAPS, xDrip+, etc.)
21+
#
22+
# When true (default):
23+
# - POST/PUT: UUID _id → extracted to 'identifier' field, new ObjectId assigned
24+
# - GET/DELETE: UUID _id queries search by 'identifier' field
25+
# When false:
26+
# - POST/PUT: UUID _id is stripped, new ObjectId assigned (UUID identity not preserved)
27+
# - GET/DELETE: UUID _id queries return empty results (no crash)
28+
# UUID_HANDLING=true

0 commit comments

Comments
 (0)