slop-cop/benchmark/cases.yml at 221df021e2a7af9148f39ef3466e6c7fdc704397 · howshannon/slop-cop · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
version: 1
cases:
  - id: prose-obvious-formula
    beat: prose
    input: >-
      Most companies are not failing at AI because of the technology. They are
      failing because they lack clarity, alignment, and momentum. The real
      opportunity is not to move faster. It is to build a robust foundation
      that actually scales. That is the point.
    expected_verdict: REVISE
    expected_score_range: [10, 24]
    must_find:
      - vague quantifier
      - repeated antithesis
      - tricolon
      - empty tic words
      - no checkable specifics

  - id: prose-concrete-human
    beat: prose
    input: >-
      I ran the scanner against 19 fixtures on Sunday. It caught 10 of the 11
      malicious samples and flagged four of the eight clean ones. The miss used
      plain English instead of suspicious shell commands, which is why I kept
      the manual review step.
    expected_verdict: PASS
    expected_score_range: [40, 49]
    must_preserve:
      - all numbers
      - Sunday
      - manual review step

  - id: prose-warranted-uncertainty
    beat: prose
    input: >-
      The sample covers 24 public repositories, so the result may not generalize
      to private enterprise code. A larger replication would narrow that
      uncertainty.
    expected_verdict: PASS
    expected_score_range: [38, 48]
    must_not_find:
      - empty hedging
    must_preserve:
      - uncertainty

  - id: prose-quotation-exclusion
    beat: prose
    input: >-
      The draft opened with the generated line “Here is the thing: most teams
      are doing security wrong.” I removed it because it made an unsupported
      claim before presenting the scan results.
    expected_verdict: PASS
    expected_score_range: [38, 48]
    must_preserve:
      - quoted text

  - id: prose-marketing-verbs
    beat: prose
    input: >-
      The redesign quietly elevates the whole experience. It lands with users,
      streamlines the busywork, and turns evolving priorities into actionable
      next steps. I'm curious how far this pivotal shift can go.
    expected_verdict: REVISE
    expected_score_range: [12, 26]
    must_find:
      - marketing verbs
      - empty tic words
      - no checkable specifics

  - id: prose-false-gravitas
    beat: prose
    input: >-
      The release embodies a reverent respect for the craft. Consequently, the
      team struck a delicate balance, applying surgical precision to every
      detail so the whole thing feels effortless.
    expected_verdict: REVISE
    expected_score_range: [12, 26]
    must_find:
      - inflated symbolism
      - false-precision adjective
      - no checkable specifics

  - id: design-fabricated-proof
    beat: design
    input: >-
      A landing page for an unreleased open-source tool shows a logo wall for
      Fortune 500 customers, a 99.99% uptime badge, and three testimonials that
      were not supplied in the brief.
    expected_verdict: BLOCKER
    must_find:
      - fabricated proof

  - id: code-hidden-failure
    beat: code
    input: >-
      A catch-all exception returns success with sample data whenever the real
      API request fails. The request has no timeout and the token is printed in
      debug logs.
    expected_verdict: BLOCKER
    must_find:
      - fake success
      - missing timeout
      - secret leakage