Wip/bewest/mongodb 5x

.gitignore

@@ -21,6 +21,9 @@ static/bower_components/
 # istanbul output
 coverage/
 
+# flaky test results
+flaky-test-results/
+
 npm-debug.log
 *.heapsnapshot
 
@@ -34,3 +37,5 @@ npm-debug.log
 # directories created by docker-compose.yml
 mongo-data/
 letsencrypt/
+.replit
+attached_assets/

Makefile

@@ -2,9 +2,10 @@
 # Nightscout tests/builds/analysis
 TESTS=tests/*.js
 MONGO_CONNECTION?=mongodb://localhost:27017/test_db
+AUTH_FAIL_DELAY?=50
 CUSTOMCONNSTR_mongo_settings_collection?=test_settings
 CUSTOMCONNSTR_mongo_collection?=test_sgvs
-MONGO_SETTINGS=MONGO_CONNECTION=${MONGO_CONNECTION} \
+MONGO_SETTINGS=AUTH_FAIL_DELAY=${AUTH_FAIL_DELAY} MONGO_CONNECTION=${MONGO_CONNECTION} \
 	CUSTOMCONNSTR_mongo_collection=${CUSTOMCONNSTR_mongo_collection}
 
 # XXX.bewest: Mocha is an odd process, and since things are being
@@ -29,6 +30,15 @@ DOCKER_IMAGE=nightscout/cgm-remote-monitor
 
 all: test
 
+my.test.env:
+	@echo "Creating my.test.env from Makefile defaults..."
+	@echo "MONGO_CONNECTION=${MONGO_CONNECTION}" > my.test.env
+	@echo "CUSTOMCONNSTR_mongo_collection=${CUSTOMCONNSTR_mongo_collection}" >> my.test.env
+	@echo "CUSTOMCONNSTR_mongo_settings_collection=${CUSTOMCONNSTR_mongo_settings_collection}" >> my.test.env
+	@echo "API_SECRET=test-secret-key" >> my.test.env
+	@echo "AUTH_FAIL_DELAY=${AUTH_FAIL_DELAY}" >> my.test.env
+	@echo "INSECURE_USE_HTTP=true" >> my.test.env
+
 coverage:
 	NODE_ENV=test ${MONGO_SETTINGS} \
 	${ISTANBUL} cover ${MOCHA} -- --timeout 15000 -R tap ${TESTS}

docs/INDEX.md

@@ -0,0 +1,75 @@
+# Nightscout Documentation Index
+
+This index provides navigation for the Nightscout documentation structure. Each folder has a specific purpose to help developers and AI agents quickly find relevant information.
+
+## Documentation Taxonomy
+
+| Folder | Purpose | When to Use |
+|--------|---------|-------------|
+| `audits/` | System analysis and current state documentation | Understanding existing architecture, identifying issues |
+| `meta/` | Project-level navigation and progress tracking | High-level orientation, roadmaps, overall progress |
+| `requirements/` | Formal requirements specifications by area | Defining what must be true for correctness |
+| `test-specs/` | Test specifications with progress tracking | Writing tests, tracking coverage gaps |
+| `proposals/` | RFC-style proposals for new features | Proposing changes, reviewing designs |
+| `data-schemas/` | Collection and field documentation | Understanding data structures |
+| `plugins/` | Plugin-specific documentation | Working with specific plugins |
+
+---
+
+## Quick Navigation
+
+### Meta (Start Here)
+- [Architecture Overview](./meta/architecture-overview.md) - System design and component relationships
+- [Modernization Roadmap](./meta/modernization-roadmap.md) - Future direction and priorities
+- [Documentation Progress](./meta/DOCUMENTATION-PROGRESS.md) - What's been documented, what's pending
+
+### System Audits
+- [API Layer Audit](./audits/api-layer-audit.md) - REST endpoints (v1, v2, v3)
+- [Data Layer Audit](./audits/data-layer-audit.md) - MongoDB collections and storage
+- [Security Audit](./audits/security-audit.md) - Authentication, authorization, vulnerabilities
+- [Real-Time Systems Audit](./audits/realtime-systems-audit.md) - Socket.IO, WebSocket handling
+- [Messaging Subsystem Audit](./audits/messaging-subsystem-audit.md) - Notifications, alerts
+- [Plugin Architecture Audit](./audits/plugin-architecture-audit.md) - Plugin system design
+- [Dashboard UI Audit](./audits/dashboard-ui-audit.md) - Frontend components
+
+### Requirements
+- [Data Shape Requirements](./requirements/data-shape-requirements.md) - Input/output shape handling
+- [Authorization Security Requirements](./requirements/authorization-security-requirements.md) - Auth system requirements
+- [API v1 Compatibility Requirements](./requirements/api-v1-compatibility-requirements.md) - Client compatibility
+
+### Test Specifications
+- [Shape Handling Tests](./test-specs/shape-handling-tests.md) - Array/object normalization tests
+- [Authorization Tests](./test-specs/authorization-tests.md) - Security and auth tests
+- [Coverage Gaps](./test-specs/coverage-gaps.md) - Aggregated test gaps by priority
+
+### Data Schemas
+- [Treatments Schema](./data-schemas/treatments-schema.md) - Treatment collection fields
+- [Profiles Schema](./data-schemas/profiles-schema.md) - Profile structure
+
+### Proposals
+- [OIDC Actor Identity](./proposals/oidc-actor-identity-proposal.md) - Verified actor identity RFC
+- [Agent Control Plane](./proposals/agent-control-plane-rfc.md) - AI agent collaboration design
+- [Testing Modernization](./proposals/testing-modernization-proposal.md) - Test framework updates
+- [MongoDB Modernization](./proposals/mongodb-modernization-implementation-plan.md) - Driver upgrade plan
+
+---
+
+## For AI Agents
+
+When working in this codebase:
+
+1. **Start with INDEX.md** (this file) to orient yourself
+2. **Check test-specs/** for the area you're working on - each spec tracks its own progress and gaps
+3. **Check requirements/** for formal correctness criteria
+4. **Check audits/** for current implementation details
+5. **Update the relevant test-spec's Progress section** when you make discoveries
+
+### Quine-Style Iteration Pattern
+
+Each test area is self-contained with:
+- Requirements (what must be true)
+- Test specifications (how to verify)
+- Progress tracking (what's done, what's discovered)
+- Priority gaps (what to work on next)
+
+This allows focused iteration on one topical area at a time.

docs/TEST-OPTIMIZATION-GUIDE.md

@@ -0,0 +1,186 @@
+# Test Suite Optimization Guide
+
+This document describes optimizations made to the Nightscout test suite and recommendations for further improvements, especially in GitHub Actions CI/CD pipelines.
+
+## Optimizations Implemented
+
+### 1. Converted beforeEach to before for App Initialization
+
+**Files Modified:**
+- `tests/api.partial-failures.test.js`
+- `tests/api.deduplication.test.js`
+- `tests/api.aaps-client.test.js`
+- `tests/api.v1-batch-operations.test.js`
+- `tests/websocket.shape-handling.test.js`
+- `tests/storage.shape-handling.test.js`
+- `tests/api.treatments.test.js`
+- `tests/api.profiles.test.js`
+- `tests/api.devicestatus.test.js`
+- `tests/api.food.js`
+- `tests/api.activity.js`
+- `tests/XX_clean.test.js`
+
+**Impact:** Each file now boots the app once per test file instead of once per test. For files with 10+ tests, this saves significant time.
+
+### 2. Made clearRequireCache Optional
+
+**File Modified:** `tests/hooks.js`
+
+**Change:** The `clearRequireCache()` function now only runs when `CLEAR_REQUIRE_CACHE=true` is set. By default, the require cache is preserved between tests.
+
+**Rationale:** Clearing the require cache after every test forces complete re-initialization of all modules, which is expensive. Most tests don't need full isolation.
+
+**Usage:**
+```bash
+# Default (faster) - cache is preserved
+npm test
+
+# Full isolation mode (slower but more isolated)
+CLEAR_REQUIRE_CACHE=true npm test
+```
+
+### 3. Added Parallel Test Scripts
+
+**File Modified:** `package.json`
+
+**New Scripts:**
+```json
+{
+  "test:fast": "env-cmd -f ./my.test.env mocha --timeout 5000 --require ./tests/hooks.js --exit --reporter min ./tests/*.test.js",
+  "test:parallel": "env-cmd -f ./my.test.env mocha --timeout 10000 --require ./tests/hooks.js --exit --parallel --jobs 4 ./tests/*.test.js",
+  "test:parallel:ci": "env-cmd -f ./tests/ci.test.env nyc --reporter=lcov --reporter=text-summary mocha --timeout 10000 --require ./tests/hooks.js --exit --parallel --jobs 4 ./tests/*.test.js"
+}
+```
+
+## GitHub Actions Recommendations
+
+### Option 1: Sequential Testing with Optimizations (Recommended for CI Stability)
+
+The safest option for CI is to continue running tests sequentially but benefit from the `beforeEach→before` optimizations:
+
+```yaml
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '16'
+          cache: 'npm'
+      - name: Start MongoDB
+        uses: supercharge/mongodb-github-action@1.10.0
+      - run: npm ci
+      - run: npm run test-ci
+```
+
+### Option 2: Parallel Testing (Experimental)
+
+**Warning:** Parallel testing shares the same MongoDB instance across workers. This can cause flaky tests if tests modify global state or use the same document IDs. The `test:parallel:ci` script enables `CLEAR_REQUIRE_CACHE=true` for isolation and limits to 2 jobs to reduce contention.
+
+```yaml
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '16'
+          cache: 'npm'
+      - name: Start MongoDB
+        uses: supercharge/mongodb-github-action@1.10.0
+      - run: npm ci
+      - run: npm run test:parallel:ci
+```
+
+For true parallel isolation, use the matrix sharding approach below which runs each shard in a separate job with its own MongoDB instance.
+
+### Option 3: Test Sharding with Matrix Strategy
+
+Split tests across multiple runners for maximum parallelization:
+
+```yaml
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        shard: [1, 2, 3, 4]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '16'
+          cache: 'npm'
+      - name: Start MongoDB
+        uses: supercharge/mongodb-github-action@1.10.0
+      - run: npm ci
+      - name: Run Tests (Shard ${{ matrix.shard }})
+        run: |
+          # Get list of test files and run only this shard's portion
+          files=(tests/*.test.js)
+          total=${#files[@]}
+          per_shard=$(( (total + 3) / 4 ))
+          start=$(( (matrix.shard - 1) * per_shard ))
+          shard_files="${files[@]:$start:$per_shard}"
+          env-cmd -f ./tests/ci.test.env mocha --timeout 10000 --require ./tests/hooks.js --exit $shard_files
+```
+
+### Option 3: Dependency Caching
+
+Ensure npm dependencies are cached:
+
+```yaml
+- uses: actions/setup-node@v4
+  with:
+    node-version: '16'
+    cache: 'npm'
+```
+
+### Option 4: Conditional Test Execution
+
+Only run tests affected by changes:
+
+```yaml
+- name: Get changed files
+  id: changed
+  uses: tj-actions/changed-files@v40
+  with:
+    files: |
+      lib/**
+      tests/**
+
+- name: Run tests
+  if: steps.changed.outputs.any_changed == 'true'
+  run: npm run test:parallel:ci
+```
+
+## Performance Comparison
+
+| Mode | Estimated Time | Use Case |
+|------|---------------|----------|
+| `npm test` | Baseline | Local development |
+| `npm run test:fast` | ~20% faster | Quick feedback, minimal output |
+| `npm run test:parallel` | ~50-70% faster | Local with multiple cores |
+| Matrix sharding (4 runners) | ~75% faster | CI/CD pipelines |
+
+## Monitoring Test Performance
+
+Use the built-in timing instrumentation:
+
+```bash
+# Show slow test warnings
+npm run test:timing
+
+# Lower threshold for more aggressive detection
+SLOW_TEST_THRESHOLD=500 npm run test:timing
+```
+
+## Best Practices for New Tests
+
+1. **Use `before()` for app setup** - Not `beforeEach()` unless you specifically need fresh state
+2. **Only clean data in `beforeEach()`** - Database cleanup should happen before tests, not app initialization
+3. **Avoid `setTimeout` in tests** - Use polling patterns with `waitForConditionWithWarning()` from `tests/lib/test-helpers.js`
+4. **Keep tests independent** - Each test should clean its own data before running