You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: docs/semgrep-assistant/metrics.md
+17-17
Original file line number
Diff line number
Diff line change
@@ -10,16 +10,16 @@ tags:
10
10
11
11
# Semgrep Assistant metrics and methodology
12
12
13
-
Semgrep's metrics for evaluating Semgrep Assistant's performance are derived from two sources:
13
+
Metrics for evaluating Semgrep Assistant's performance are derived from two sources:
14
14
15
15
-**User feedback** on Assistant recommendations within the product
16
16
-**Internal triage and benchmarking** conducted by Semgreps security research team
17
17
18
18
This methodology ensures that Assistant is evaluated from both a user's and expert's perspective. This gives Semgrep's product and engineering teams a holistic view into Assistant's real-world performance.
19
19
20
-
## User feedback (real-world dataset)
20
+
## User feedback
21
21
22
-
User feedback shows the aggregated and anonymized performance of Assistant across **more than 1000 customers**, providing a comprehensive real-world dataset.
22
+
User feedback shows the aggregated and anonymized performance of Assistant across **more than 1000 customers**, providing a comprehensive **real-world dataset**.
23
23
24
24
Users are prompted in-line to "thumbs up" or "thumbs down" Assistant suggestions as they receive Assistant suggestions in their PR or MR. This ensures that sampling bias is reduced, as both developers and AppSec engineers can provide feedback.
25
25
@@ -28,23 +28,27 @@ Users are prompted in-line to "thumbs up" or "thumbs down" Assistant suggestions
28
28
<table>
29
29
<tr>
30
30
<td>Customers in dataset</td>
31
-
<td><b>1000+</b></td>
31
+
<td><strong>1000+</strong></td>
32
32
</tr>
33
33
<tr>
34
34
<td>Findings analyzed</td>
35
-
<td><b>250,000+</b></td>
35
+
<td><strong>250,000+</strong></td>
36
+
</tr>
37
+
<tr>
38
+
<td>Average reduction in findings[^1]</td>
39
+
<td><strong>20%</strong></td>
36
40
</tr>
37
41
<tr>
38
42
<td>Human-agree rate</td>
39
-
<td><b>92%</b></td>
43
+
<td><strong>92%</strong></td>
40
44
</tr>
41
45
<tr>
42
46
<td>Median time to resolution</td>
43
-
<td><b>22% faster than baseline</b></td>
47
+
<td><strong>22% faster than baseline</strong></td>
44
48
</tr>
45
49
<tr>
46
50
<td>Average time saved per finding</td>
47
-
<td><b>30 minutes</b></td>
51
+
<td><strong>30 minutes</strong></td>
48
52
</tr>
49
53
</table>
50
54
@@ -57,24 +61,20 @@ Internal benchmarks for Assistant run on the same dataset used by Semgrep's secu
57
61
<table>
58
62
<tr>
59
63
<td>Findings analyzed</td>
60
-
<td><b>2000+</b></td>
61
-
</tr>
62
-
<tr>
63
-
<td>Average reduction in findings[^1]</td>
64
-
<td><b>20%</b></td>
64
+
<td><strong>2000+</strong></td>
65
65
</tr>
66
66
<tr>
67
67
<td>False positive confidence rate[^2]</td>
68
-
<td><b>96%</b></td>
68
+
<td><strong>96%</strong></td>
69
69
</tr>
70
70
<tr>
71
71
<td>Remediation guidance confidence rate[^3]</td>
72
-
<td><b>80%</b></td>
72
+
<td><strong>80%</strong></td>
73
73
</tr>
74
74
</table>
75
75
76
-
[^1]:The average % of SAST findings that Assistant filters out as noise.
76
+
[^1]:The average % of SAST findings that Assistant filters out as noise.
77
77
78
78
[^2]:False positive confidence rate measures how often Assistant is correct when it identifies a false positive. **A high confidence rate means users can trust when Assistant identifies a false positive - it does not mean that Assistant catches all false positives.**
79
79
80
-
[^3]:Remediation guidance is rated on a binary scale of "helpful" / "not helpful".
80
+
[^3]:Remediation guidance is rated on a binary scale of "helpful" / "not helpful".
0 commit comments