Skip to content

Commit 423e22a

Browse files
apankov1claude
andauthored
feat: add 3 remaining slop rules (#17) (#18)
## Summary Completes the production audit rule set from #14. Adds the 3 remaining rules: - **`vacuous_property`** (should-fail): Detects `fc.property` callbacks with `return true` paths that bypass assertions (2a), and `fc.constant`-only generators with zero input variation (2b) - **`no_production_call`** (should-fail): Detects tests that call no imported production function — only builtins, operators, or language guarantees. Includes `parseImports()` helper for import tracking. - **`impossible_assertion`** (should-fail): Detects assertions that are mathematically impossible to fail, e.g. `expect(x.length).toBeGreaterThanOrEqual(0)` Updates total rule count from 15 to 18 across presets, SKILL.md, and pattern reference. ## Test plan - [x] 17 new tests covering all 3 rules + `parseImports` helper - [x] All 369 tests pass across 9 skills - [x] Biome check clean - [x] Preset count assertions updated from 15 to 18 - [x] SKILL.md Violation Rules table and What To Protect table updated - [x] references/patterns.md updated with before/after examples (patterns 16-18) Closes #17 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 939b6b1 commit 423e22a

4 files changed

Lines changed: 649 additions & 10 deletions

File tree

skills/slop-test-detector/SKILL.md

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
name: slop-test-detector
3-
description: "Static analyzer that detects 15 slop patterns in test code — tests that compile and pass but catch zero bugs."
3+
description: "Static analyzer that detects 18 slop patterns in test code — tests that compile and pass but catch zero bugs."
44
---
55

66
# Slop Test Detector
@@ -36,6 +36,9 @@ Before auditing or generating tests, identify which slop patterns apply:
3636
| Tests vary their inputs | Do sibling tests exercise different code paths? | `no_input_variation`, `duplicate_assertion_set` |
3737
| Assertions test computed values | Are assertions checking results, not echoing construction literals? | `literal_roundtrip`, `schema_success_only` |
3838
| Assertions always execute | Can the test pass without any assertion running? | `conditional_assertion` |
39+
| Property tests are meaningful | Do fc.property callbacks assert on all paths with varied inputs? | `vacuous_property` |
40+
| Tests exercise production code | Does the test call an imported function, not just builtins? | `no_production_call` |
41+
| Assertions can actually fail | Is the assertion mathematically capable of failing? | `impossible_assertion` |
3942

4043
---
4144

@@ -64,6 +67,7 @@ import {
6467
formatReport,
6568
formatReportJSON,
6669
getPreset,
70+
parseImports,
6771
parseTestFile,
6872
} from './slop-detector.ts';
6973
```
@@ -76,9 +80,9 @@ Three built-in presets control which rules are active:
7680

7781
| Preset | Rules | Default threshold | Use case |
7882
|--------|-------|-------------------|----------|
79-
| `balanced` | 10 rules (no defect-comment rules) | 80 | **Default.** Conservative, low noise |
80-
| `strict` | All 12 rules | 90 | Teams that enforce `// Defect:` comments |
81-
| `advisory` | All 12 rules | 0 | Report everything, fail nothing |
83+
| `balanced` | 16 rules (no defect-comment rules) | 80 | **Default.** Conservative, low noise |
84+
| `strict` | All 18 rules | 90 | Teams that enforce `// Defect:` comments |
85+
| `advisory` | All 18 rules | 0 | Report everything, fail nothing |
8286

8387
```typescript
8488
import { getPreset } from './slop-detector.ts';
@@ -188,6 +192,9 @@ score = max(0, round(100 × (1 - weightedFindings / testCount)))
188192
| `literal_roundtrip` | Assertion compares `obj.field` to the same literal used to construct `obj` | should-fail | on |
189193
| `schema_success_only` | `safeParse()` result checked for `.success` but never `.data` or `.error.issues` | should-fail | on |
190194
| `conditional_assertion` | All assertions are inside `if`/`switch` blocks — test may silently pass | must-fail | on |
195+
| `vacuous_property` | `fc.property` callback has `return true` path with zero assertions, or all generators are `fc.constant` | should-fail | on |
196+
| `no_production_call` | Test body calls no imported production function — only builtins or language guarantees | should-fail | on |
197+
| `impossible_assertion` | Assertion is mathematically impossible to fail (e.g., `.length >= 0`) | should-fail | on |
191198

192199
**opt-in** rules are only active in the `strict` preset. Use `getPreset('strict')` or add them to a custom `enabledRules` set.
193200

skills/slop-test-detector/references/patterns.md

Lines changed: 99 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Slop Patterns Reference
22

3-
All 15 slop patterns with before (slop) and after (fixed) examples.
3+
All 18 slop patterns with before (slop) and after (fixed) examples.
44

55
---
66

@@ -352,3 +352,101 @@ it("handles results", () => {
352352
assert.equal(results[0].status, "ok");
353353
});
354354
```
355+
356+
---
357+
358+
### 16. vacuous_property
359+
360+
**Before (slop)**`return true` path with zero assertions (2a):
361+
```typescript
362+
it('priority correctness', () => {
363+
fc.assert(
364+
fc.property(gameKindGen, eventTypeGen, (gameKind, eventType) => {
365+
if (hasGameSpecific && isPlatformEvent) {
366+
expect(result).toEqual(gameSpecificMeta);
367+
return true;
368+
}
369+
return true; // most inputs hit this path — 0 assertions
370+
}),
371+
);
372+
});
373+
```
374+
375+
**Before (slop)** — zero-variation generators (2b):
376+
```typescript
377+
it('default constructor', () => {
378+
fc.assert(
379+
fc.property(fc.constant(undefined), (_ignored) => {
380+
const chain = new EventHashChain();
381+
expect(chain.getSemanticHash()).toBeUndefined();
382+
// fc.constant = same test every run
383+
}),
384+
);
385+
});
386+
```
387+
388+
**After (fixed)** — assert on all paths with real generators:
389+
```typescript
390+
it('priority correctness', () => {
391+
fc.assert(
392+
fc.property(gameKindGen, eventTypeGen, (gameKind, eventType) => {
393+
const result = resolvePriority(gameKind, eventType);
394+
expect(result).toBeDefined();
395+
expect(typeof result.priority).toBe('number');
396+
}),
397+
);
398+
});
399+
```
400+
401+
---
402+
403+
### 17. no_production_call
404+
405+
**Before (slop)** — test only exercises builtins, no production function called:
406+
```typescript
407+
it('expert-level weights must always yield negative net score', () => {
408+
fc.assert(
409+
fc.property(
410+
fc.integer({ min: -100, max: -50 }),
411+
fc.integer({ min: 20, max: 49 }),
412+
(breakingPenalty, creationBonus) => {
413+
const netScore = breakingPenalty + creationBonus;
414+
return netScore < 0; // pure arithmetic — no production function
415+
},
416+
),
417+
);
418+
});
419+
```
420+
421+
**After (fixed)** — call the actual scoring function:
422+
```typescript
423+
it('expert-level weights must always yield negative net score', () => {
424+
fc.assert(
425+
fc.property(
426+
fc.integer({ min: -100, max: -50 }),
427+
fc.integer({ min: 20, max: 49 }),
428+
(breakingPenalty, creationBonus) => {
429+
const score = computeNetScore(breakingPenalty, creationBonus);
430+
return score < 0;
431+
},
432+
),
433+
);
434+
});
435+
```
436+
437+
---
438+
439+
### 18. impossible_assertion
440+
441+
**Before (slop)** — assertion that is mathematically impossible to fail:
442+
```typescript
443+
expect(Object.keys(aliasToCanonical).length).toBeGreaterThanOrEqual(0);
444+
// .length is always >= 0 — this CANNOT fail
445+
```
446+
447+
**After (fixed)** — assert on a specific expected length:
448+
```typescript
449+
expect(Object.keys(aliasToCanonical).length).toBeGreaterThanOrEqual(1);
450+
// or assert the exact count:
451+
expect(Object.keys(aliasToCanonical)).toHaveLength(5);
452+
```

0 commit comments

Comments
 (0)