|
| 1 | +# Test Suite Optimization Guide |
| 2 | + |
| 3 | +This document describes optimizations made to the Nightscout test suite and recommendations for further improvements, especially in GitHub Actions CI/CD pipelines. |
| 4 | + |
| 5 | +## Optimizations Implemented |
| 6 | + |
| 7 | +### 1. Converted beforeEach to before for App Initialization |
| 8 | + |
| 9 | +**Files Modified:** |
| 10 | +- `tests/api.partial-failures.test.js` |
| 11 | +- `tests/api.deduplication.test.js` |
| 12 | +- `tests/api.aaps-client.test.js` |
| 13 | +- `tests/api.v1-batch-operations.test.js` |
| 14 | +- `tests/websocket.shape-handling.test.js` |
| 15 | +- `tests/storage.shape-handling.test.js` |
| 16 | +- `tests/api.treatments.test.js` |
| 17 | +- `tests/api.profiles.test.js` |
| 18 | +- `tests/api.devicestatus.test.js` |
| 19 | +- `tests/api.food.js` |
| 20 | +- `tests/api.activity.js` |
| 21 | +- `tests/XX_clean.test.js` |
| 22 | + |
| 23 | +**Impact:** Each file now boots the app once per test file instead of once per test. For files with 10+ tests, this saves significant time. |
| 24 | + |
| 25 | +### 2. Made clearRequireCache Optional |
| 26 | + |
| 27 | +**File Modified:** `tests/hooks.js` |
| 28 | + |
| 29 | +**Change:** The `clearRequireCache()` function now only runs when `CLEAR_REQUIRE_CACHE=true` is set. By default, the require cache is preserved between tests. |
| 30 | + |
| 31 | +**Rationale:** Clearing the require cache after every test forces complete re-initialization of all modules, which is expensive. Most tests don't need full isolation. |
| 32 | + |
| 33 | +**Usage:** |
| 34 | +```bash |
| 35 | +# Default (faster) - cache is preserved |
| 36 | +npm test |
| 37 | + |
| 38 | +# Full isolation mode (slower but more isolated) |
| 39 | +CLEAR_REQUIRE_CACHE=true npm test |
| 40 | +``` |
| 41 | + |
| 42 | +### 3. Added Parallel Test Scripts |
| 43 | + |
| 44 | +**File Modified:** `package.json` |
| 45 | + |
| 46 | +**New Scripts:** |
| 47 | +```json |
| 48 | +{ |
| 49 | + "test:fast": "env-cmd -f ./my.test.env mocha --timeout 5000 --require ./tests/hooks.js --exit --reporter min ./tests/*.test.js", |
| 50 | + "test:parallel": "env-cmd -f ./my.test.env mocha --timeout 10000 --require ./tests/hooks.js --exit --parallel --jobs 4 ./tests/*.test.js", |
| 51 | + "test:parallel:ci": "env-cmd -f ./tests/ci.test.env nyc --reporter=lcov --reporter=text-summary mocha --timeout 10000 --require ./tests/hooks.js --exit --parallel --jobs 4 ./tests/*.test.js" |
| 52 | +} |
| 53 | +``` |
| 54 | + |
| 55 | +## GitHub Actions Recommendations |
| 56 | + |
| 57 | +### Option 1: Sequential Testing with Optimizations (Recommended for CI Stability) |
| 58 | + |
| 59 | +The safest option for CI is to continue running tests sequentially but benefit from the `beforeEach→before` optimizations: |
| 60 | + |
| 61 | +```yaml |
| 62 | +jobs: |
| 63 | + test: |
| 64 | + runs-on: ubuntu-latest |
| 65 | + steps: |
| 66 | + - uses: actions/checkout@v4 |
| 67 | + - uses: actions/setup-node@v4 |
| 68 | + with: |
| 69 | + node-version: '16' |
| 70 | + cache: 'npm' |
| 71 | + - name: Start MongoDB |
| 72 | + uses: supercharge/mongodb-github-action@1.10.0 |
| 73 | + - run: npm ci |
| 74 | + - run: npm run test-ci |
| 75 | +``` |
| 76 | +
|
| 77 | +### Option 2: Parallel Testing (Experimental) |
| 78 | +
|
| 79 | +**Warning:** Parallel testing shares the same MongoDB instance across workers. This can cause flaky tests if tests modify global state or use the same document IDs. The `test:parallel:ci` script enables `CLEAR_REQUIRE_CACHE=true` for isolation and limits to 2 jobs to reduce contention. |
| 80 | + |
| 81 | +```yaml |
| 82 | +jobs: |
| 83 | + test: |
| 84 | + runs-on: ubuntu-latest |
| 85 | + steps: |
| 86 | + - uses: actions/checkout@v4 |
| 87 | + - uses: actions/setup-node@v4 |
| 88 | + with: |
| 89 | + node-version: '16' |
| 90 | + cache: 'npm' |
| 91 | + - name: Start MongoDB |
| 92 | + uses: supercharge/mongodb-github-action@1.10.0 |
| 93 | + - run: npm ci |
| 94 | + - run: npm run test:parallel:ci |
| 95 | +``` |
| 96 | + |
| 97 | +For true parallel isolation, use the matrix sharding approach below which runs each shard in a separate job with its own MongoDB instance. |
| 98 | + |
| 99 | +### Option 3: Test Sharding with Matrix Strategy |
| 100 | + |
| 101 | +Split tests across multiple runners for maximum parallelization: |
| 102 | + |
| 103 | +```yaml |
| 104 | +jobs: |
| 105 | + test: |
| 106 | + runs-on: ubuntu-latest |
| 107 | + strategy: |
| 108 | + fail-fast: false |
| 109 | + matrix: |
| 110 | + shard: [1, 2, 3, 4] |
| 111 | + steps: |
| 112 | + - uses: actions/checkout@v4 |
| 113 | + - uses: actions/setup-node@v4 |
| 114 | + with: |
| 115 | + node-version: '16' |
| 116 | + cache: 'npm' |
| 117 | + - name: Start MongoDB |
| 118 | + uses: supercharge/mongodb-github-action@1.10.0 |
| 119 | + - run: npm ci |
| 120 | + - name: Run Tests (Shard ${{ matrix.shard }}) |
| 121 | + run: | |
| 122 | + # Get list of test files and run only this shard's portion |
| 123 | + files=(tests/*.test.js) |
| 124 | + total=${#files[@]} |
| 125 | + per_shard=$(( (total + 3) / 4 )) |
| 126 | + start=$(( (matrix.shard - 1) * per_shard )) |
| 127 | + shard_files="${files[@]:$start:$per_shard}" |
| 128 | + env-cmd -f ./tests/ci.test.env mocha --timeout 10000 --require ./tests/hooks.js --exit $shard_files |
| 129 | +``` |
| 130 | + |
| 131 | +### Option 3: Dependency Caching |
| 132 | + |
| 133 | +Ensure npm dependencies are cached: |
| 134 | + |
| 135 | +```yaml |
| 136 | +- uses: actions/setup-node@v4 |
| 137 | + with: |
| 138 | + node-version: '16' |
| 139 | + cache: 'npm' |
| 140 | +``` |
| 141 | + |
| 142 | +### Option 4: Conditional Test Execution |
| 143 | + |
| 144 | +Only run tests affected by changes: |
| 145 | + |
| 146 | +```yaml |
| 147 | +- name: Get changed files |
| 148 | + id: changed |
| 149 | + uses: tj-actions/changed-files@v40 |
| 150 | + with: |
| 151 | + files: | |
| 152 | + lib/** |
| 153 | + tests/** |
| 154 | +
|
| 155 | +- name: Run tests |
| 156 | + if: steps.changed.outputs.any_changed == 'true' |
| 157 | + run: npm run test:parallel:ci |
| 158 | +``` |
| 159 | + |
| 160 | +## Performance Comparison |
| 161 | + |
| 162 | +| Mode | Estimated Time | Use Case | |
| 163 | +|------|---------------|----------| |
| 164 | +| `npm test` | Baseline | Local development | |
| 165 | +| `npm run test:fast` | ~20% faster | Quick feedback, minimal output | |
| 166 | +| `npm run test:parallel` | ~50-70% faster | Local with multiple cores | |
| 167 | +| Matrix sharding (4 runners) | ~75% faster | CI/CD pipelines | |
| 168 | + |
| 169 | +## Monitoring Test Performance |
| 170 | + |
| 171 | +Use the built-in timing instrumentation: |
| 172 | + |
| 173 | +```bash |
| 174 | +# Show slow test warnings |
| 175 | +npm run test:timing |
| 176 | +
|
| 177 | +# Lower threshold for more aggressive detection |
| 178 | +SLOW_TEST_THRESHOLD=500 npm run test:timing |
| 179 | +``` |
| 180 | + |
| 181 | +## Best Practices for New Tests |
| 182 | + |
| 183 | +1. **Use `before()` for app setup** - Not `beforeEach()` unless you specifically need fresh state |
| 184 | +2. **Only clean data in `beforeEach()`** - Database cleanup should happen before tests, not app initialization |
| 185 | +3. **Avoid `setTimeout` in tests** - Use polling patterns with `waitForConditionWithWarning()` from `tests/lib/test-helpers.js` |
| 186 | +4. **Keep tests independent** - Each test should clean its own data before running |
0 commit comments