Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update info re: Assistant memories and noise filtering #1903

Merged
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/semgrep-appsec-platform/dashboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ _**Figure**. Secure guardrails pane. Hover over the charts to view data for that
| Chart | Description |
| ------- | ------ |
| Secure guardrails adoption | Percent of new findings shown to developers over the specified time period. An upward or stable trend is better. |
| Guardrails activity | This chart displays a breakdown of the status of findings shown to developers; whether they were ignored, fixed, or remained open. A greater **Fixed** value is better. |
| Guardrails activity | This chart displays a breakdown of the status of findings shown to developers; whether they were ignored, fixed, remained open, or [filtered by Assistant](/semgrep-assistant/overview#noise-filtering). A greater **Fixed** value is better. |

## Most findings by project

Expand Down
67 changes: 53 additions & 14 deletions docs/semgrep-assistant/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,9 @@ Semgrep Assistant requires [read access to your code in GitHub](https://docs.git
4. When prompted, click **Continue** to allow redirection to GitHub to finalize app creation. Follow the instructions to finish creating and installing a private `semgrep-app`.
4. You are redirected to Semgrep AppSec Platform's **Source Code Managers** page. Navigate back to the **Deployment** page. Under the **Assistant** section, verify that all of the features are enabled:
1. **Allow code snippets in AI prompts**: Required for Semgrep to auto-triage findings, provide AI remediation guidance, and tag findings with code context.
2. **Auto-triage for Code**: Enable notifications whenever Assistant suggests that a finding may be safe to ignore. You can include notifications in your PR and MR comments, or you can receive them through Slack notifications.
3. **Autofix suggestions for Code**: Enable Assistant-generated autofix suggestions in comments from Assistant. You can also set the minimum confidence level for Assistant-written fixes if the Semgrep rule doesn't include a human-written autofix.
![Semgrep Assistant toggle location](/img/semgrep-assistant-enable.png#md-width)
2. **Weekly priority emails**: Enable weekly emails to all organization admins with information on Assistant's top three backlog tasks across all findings.
3. **Noise filter for Code PR/MR comments**: Enable the filtering of findings flagged as false positives. You can choose to suppress any PR or MR comments Semgrep might push, or you can choose to show developers information regarding false positives using PR or MR comments.
4. **Remediation**: Enable Assistant-generated autofix suggestions in comments from Assistant. You can also set the minimum confidence level for Assistant-written fixes if the Semgrep rule doesn't include a human-written autofix.

</TabItem>

Expand All @@ -82,24 +82,36 @@ Semgrep Assistant requires the **API scope** to run in both GitLab SaaS and GitL
1. Follow the on-screen instructions to complete the setup process.
2. Navigate back to the **Deployment** page. Under the **Assistant** section, verify that all of the features are enabled:
1. **Allow code snippets in AI prompts**: Required for Semgrep to auto-triage findings, provide AI remediation guidance, and tag findings with code context.
2. **Autofix suggestions for Code**: Enable autofix suggestions in comments from Assistant. You can also set the minimum confidence level for Assistant-written fixes if the Semgrep rule doesn't include a human-written autofix.
3. **Auto-triage for Code**: Enable notifications whenever Assistant suggests that a finding may be safe to ignore. You can include notifications in your PR and MR comments, or you can receive them through Slack notifications.
![Semgrep Assistant toggle location](/img/semgrep-assistant-enable.png#md-width)
2. **Weekly priority emails**: Enable weekly emails to all organization admins with information on Assistant's top three backlog tasks across all findings.
3. **Noise filter for Code PR/MR comments**: Enable the filtering of findings flagged as false positives. You can choose to suppress any PR or MR comments Semgrep might push, or you can choose to show developers information regarding false positives using PR or MR comments.
4. **Remediation**: Enable Assistant-generated autofix suggestions in comments from Assistant. You can also set the minimum confidence level for Assistant-written fixes if the Semgrep rule doesn't include a human-written autofix.

</TabItem>
</Tabs>

### Enable autofix suggestions
### Enable remediation

Assistant autofix allows you to receive AI-generated code snippets to remediate true positives. Perform the following to enable it:
Assistant remediation allows you to receive AI-generated code snippets for true positives. Perform the following to enable it:

1. Sign in to Semgrep AppSec Platform, and navigate to **Settings > Deployment**.
2. In the **Assistant** section, click the *Autofix suggestions for Code** <i class="fa-solid fa-toggle-large-on"></i> if it is not yet enabled.
2. In the **Assistant** section, click the **Remediation** <i class="fa-solid fa-toggle-large-on"></i> if it is not yet enabled.
3. *Optional*: Select a **confidence level** in the drop-down box. This value determines the level of quality at which the autofix code appears as a suggestion. A lower confidence level means that Semgrep Assistant displays the autofix suggestion even when the code quality may be incorrect.
:::tip
Semgrep recommends setting a low confidence level since even incorrect suggestions may be useful starting points for triage and remediation.
:::

### Enable noise filtering

Assistant is over 95% accurate in categorizing Semgrep Code findings as false positives, so you can minimize the number of findings shown by enabling **Noise filter for Code PR/MR comments**. To do so:

1. Sign in to Semgrep AppSec Platform, and navigate to **Settings > Deployment**.
2. In the **Assistant** section, click the **Noise filter for Code PR/MR comments** <i class="fa-solid fa-toggle-large-on"></i> if it is not yet enabled.
3. Select whether you want to enable PR or MR comments:
1. **Don’t leave a PR/MR comment**: Hide Semgrep’s comments on findings that are likely to be false positives. These findings are available for security review on the **Code > Production backlog** page. Comments still appear for rules in [**Block** mode](/semgrep-code/policies#block-a-pr-or-mr-through-rule-modes).
2. **Include a notification in the PR/MR comment**: Show developers likely false positive findings in PR/MR comments, but include a note explaining why Assistant thinks the finding may be safe to ignore.

Findings filtered out by Assistant can be reviewed at any time in Semgrep by going to **Code > Production backlog**. Semgrep also allows you to agree with the filtering to close the finding or disagree to reopen.

### Enable auto-triage

If [auto-triage](/semgrep-assistant/overview/#auto-triage), which allows you to get notifications whenever Assistant indicates a finding may be safe to ignore, isn't enabled for your deployment, you can do so as follows:
Expand All @@ -118,16 +130,25 @@ Semgrep Assistant messages only appear in your PR comments for rules that are se
![ Policies modes](/img/semgrep-assistant-comment.png#md-width)
* You have selected PR/MR comments in **Semgrep AppSec Platform > Settings > Deployment** in the **Code** section.

### Enable weekly priority emails

If [weekly priority emails](/semgrep-assistant/overview/#weekly-priority-emails), which allows organization admins to receive information on top backlog tasks according to Assistant, isn't enabled for your deployment, you can do so as follows:

1. Sign in to Semgrep AppSec Platform, and navigate to **Settings > Deployment**.
2. In the **Assistant** section, click the **Weekly priority emails** <i class="fa-solid fa-toggle-large-on"></i> if it is not yet enabled.

## Add Memories (beta)

Assistant Memories allows AppSec teams and developers to tailor Assistant's remediation guidance to their organization's standards and defaults on a per-project, per-rule basis. Whenever Assistant gives a suggested fix, you can provide feedback by adding custom instructions.
Assistant Memories allows admins to tailor Assistant's remediation guidance to their organization's standards and defaults on a per-project, per-rule basis. You can provide feedback by adding custom instructions whenever Assistant gives a suggested fix.

Memories are enabled by default for all Assistant users.
Memories are enabled by default for all organizations with Assistant enabled.

To add a memory:
### Add memory based on Assistant's suggested fix
khorne3 marked this conversation as resolved.
Show resolved Hide resolved

1. Identify the specific instance of **Assistant's suggested fix** that you want to modify. These can be found in the finding details page or in the PR or MR comment.
2. Click **Customize fix** to open an input box, and enter your preferred remediation approaches and secure defaults for the project. The suggestion you provide can be as general as "Use AWS Secrets Manager to manage secrets."
To add a memory modifying a suggested fix presented by Assistant:

1. Identify the specific instance of **Assistant's suggested fix** that you want to modify. These can be found on the finding details page or in the PR or MR comment.
2. Click **Customize fix** to open an input box, and enter your preferred remediation approaches and secure defaults for the project. Your suggestion can be as general as "Use AWS Secrets Manager to manage secrets."
![Assistant’s suggested fix for a hardcoded secret in the user’s code](/img/memories-3.png#md-width)
***Figure***. Assistant’s suggested fix for a hardcoded secret in the user’s code.
3. Click **Save and regenerate**.
Expand All @@ -139,6 +160,24 @@ To add a memory:

While Assistant Memories is in **public beta**, memories are scoped to remediation guidance on a per-project and per-rule basis. A saved memory only affects future guidance for findings triggered by the same rule in the same project.

### Add memory during triage

If you identify findings that are safe to ignore and write triage notes indicating why this is so, Assistant can store this information as a memory and use it when assessing whether a similar finding should be shown to developers in the future. Assistant also takes that memory, reanalyzes similar findings in your backlog, and suggests issues that may be safe to close.

To add a memory during triage:

1. Identify the specific finding you want to modify, and open up its finding details page.
2. Change the status of the finding to **Ignored**, and optionally, select an **Ignore reason**.
3. Click **Ignore & add memory**.
4. In the **Create memory** pop-up window:
khorne3 marked this conversation as resolved.
Show resolved Hide resolved
1. Provide your preferred remediation approaches and secure defaults for the project. Your suggestion can be as general as "Use AWS Secrets Manager to manage secrets."
khorne3 marked this conversation as resolved.
Show resolved Hide resolved
2. Provide the **Projects** to which this memory should be applied.
3. Provide the **Rules** to which this memory should be applied.
4. Select the **Apply to <span className="placeholder">X</span> existing findings in scope** box if you would like Semgrep to apply this memory to any existing findings automatically.
5. Click **Add memory** to save your changes.
![User-provided instructions for generating a memory during the triage process.](/img/triage-memories.png#md-width)
***Figure***. User-provided instructions for generating a memory during the triage process.

### Remove Memories

1. Sign in to [Semgrep AppSec Platform](https://semgrep.dev/login?return_path=/manage/projects) and navigate to [<i class="fa-solid fa-gear"></i> **Settings > Deployment**](https://semgrep.dev/orgs/-/settings).
Expand Down
11 changes: 8 additions & 3 deletions docs/semgrep-assistant/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,10 +91,15 @@ Auto-triage recommendations are available in Semgrep AppSec Platform's **Finding
![Semgrep Assistant in the filtered Findings page](/img/semgrep-assistant-autotriage-findings.png#md-width)
_**Figure.** Semgrep Assistant auto-triage in the Findings page._

Assistant's suggestions to ignore findings are also surfaced in PR or MR comments, so developers can triage an issue without switching contexts, as well as being sent through Slack.
Assistant's suggestions to ignore findings are also surfaced in PR or MR comments, so developers can triage an issue without switching contexts.
khorne3 marked this conversation as resolved.
Show resolved Hide resolved

![Semgrep Assistant in a Slack notification](/img/semgrep-assistant-slack.png#md-width)
_**Figure.** Semgrep Assistant auto-triage in a Slack notification._
### Noise filtering

Assistant is over 95% accurate in categorizing Semgrep Code findings as false positives, so you can minimize the number of findings shown to users by hiding those flagged by Assistant as likely to be false positives. Findings filtered out by Assistant can be reviewed at any time in Semgrep. Semgrep also allows you to agree with the filtering to close the finding or disagree to reopen.
khorne3 marked this conversation as resolved.
Show resolved Hide resolved

### Weekly priority emails

Semgrep sends weekly emails with information on Assistant's top three backlog tasks across all findings. Unlike other Assistant features, these suggestions can include information for all Semgrep products that you have enabled. The emails are sent out on Monday to all organization admins.

### Memories (beta)

Expand Down
Binary file removed static/img/semgrep-assistant-enable.png
Binary file not shown.
Binary file removed static/img/semgrep-assistant-slack.png
Binary file not shown.
Binary file added static/img/triage-memories.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading