-
Notifications
You must be signed in to change notification settings - Fork 82
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[KB]: Add self hosted config options for Azure Pipelines + Ubuntu (#1887
) --------- Co-authored-by: s-santillan <[email protected]>
- Loading branch information
1 parent
7b37c64
commit 82cd687
Showing
5 changed files
with
165 additions
and
33 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,134 @@ | ||
| --- | ||
| tags: | ||
| - Azure Pipelines | ||
| description: Run Semgrep on self-hosted Ubuntu runners in Azure DevOps. | ||
| --- | ||
| import AzureVariables from "/src/components/procedure/_set-env-vars-azure.mdx" | ||
|
|
||
| # Semgrep with self-hosted Ubuntu runners in Azure Pipelines | ||
|
|
||
| Semgrep provides a [sample configuration for Azure-hosted runners](/docs/semgrep-ci/sample-ci-configs#azure-pipelines). If you use self-hosted Ubuntu Linux runners, you have significantly more control over their configuration, but as a result, they require additional preparation and configuration to run Semgrep. | ||
|
|
||
| This guide adds two approaches to configuring self-hosted runners that use Ubuntu (the default self-hosted option for Azure DevOps Linux runners): | ||
|
|
||
| * [Using pipx](#using-pipx) | ||
| * [Using pip with a virtual environment](#using-pip-with-a-virtual-environment) | ||
|
|
||
| ## Using pipx | ||
|
|
||
| While the sample configuration uses `pip`, this approach uses `pipx`, which avoids issues with system-managed Python vs user-installed Python. | ||
|
|
||
| ### Prepare your runner | ||
|
|
||
| Access the runner and execute the following commands: | ||
|
|
||
| ```bash | ||
| $ sudo apt update | ||
| $ sudo apt install pipx | ||
| $ pipx ensurepath | ||
| ``` | ||
|
|
||
| After completing the commands: | ||
|
|
||
| 1. Start a new shell session, so that the changes from `pipx ensurepath` are available. | ||
| 2. Ensure the [Azure DevOps agent](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/linux-agent?view=azure-devops) is set up and running. | ||
|
|
||
| ### Create your configuration | ||
|
|
||
| 1. Follow the steps provided in the [sample configuration for Azure-hosted runners](/docs/semgrep-ci/sample-ci-configs#azure-pipelines). | ||
| 2. Add the following snippet to the `azure-pipelines.yml` for the repository. | ||
|
|
||
| ```yaml | ||
| variables: | ||
| - group: Semgrep_Variables | ||
|
|
||
| pool: | ||
| name: Default | ||
|
|
||
| steps: | ||
| - checkout: self | ||
| clean: true | ||
| fetchDepth: 20 | ||
| persistCredentials: true | ||
| - script: | | ||
| pipx install semgrep | ||
| if [ $(Build.SourceBranchName) = "master" ]; then | ||
| echo "Semgrep full scan" | ||
| semgrep ci | ||
| elif [ $(System.PullRequest.PullRequestId) -ge 0 ]; then | ||
| echo "Semgrep diff scan" | ||
| git fetch origin master:origin/master | ||
| export SEMGREP_PR_ID=$(System.PullRequest.PullRequestId) | ||
| export SEMGREP_BASELINE_REF='origin/master' | ||
| semgrep ci | ||
| fi | ||
| env: | ||
| SEMGREP_APP_TOKEN: $(SEMGREP_APP_TOKEN) | ||
| ``` | ||
| :::info Customizing the configuration | ||
| * If your self-hosted runner [agent pool](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/pools-queues?view=azure-devops&tabs=yaml%2Cbrowser) has a different name, update the `name` key under `pool` to match the desired agent pool. | ||
| * If your default branch is not called `master`, update the references to `master` to match the name of your default branch. | ||
| ::: | ||
|
|
||
| <AzureVariables /> | ||
|
|
||
| ## Using pip with a virtual environment | ||
|
|
||
| ### Prepare your runner | ||
|
|
||
| This approach uses built-in Azure DevOps tasks, including `UsePythonVersion` and `Bash`, and uses a virtual environment to install `pip`, another approach that prevents issues with system-managed Python vs user-installed Python. | ||
|
|
||
| 1. Ensure you have a pre-installed and configured compatible version of Python 3, following [the instructions for UsePythonVersion for self-hosted runners](https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/use-python-version-v0?view=azure-pipelines#how-can-i-configure-a-self-hosted-agent-to-use-this-task). | ||
| 2. Ensure the [Azure DevOps agent](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/linux-agent?view=azure-devops) is set up and running. | ||
|
|
||
| ### Create your configuration | ||
|
|
||
| Add the following snippet to the `azure-pipelines.yml` for the repository. | ||
|
|
||
|
|
||
| ```yaml | ||
| variables: | ||
| - group: Semgrep_Variables | ||
| pool: | ||
| name: Default | ||
| steps: | ||
| - checkout: self | ||
| clean: true | ||
| persistCredentials: true | ||
| - task: UsePythonVersion@0 | ||
| displayName: 'Use Python 3.12' | ||
| inputs: | ||
| versionSpec: 3.12 | ||
| - task: Bash@3 | ||
| env: | ||
| SEMGREP_APP_TOKEN: $(SEMGREP_APP_TOKEN) | ||
| inputs: | ||
| targetType: 'inline' | ||
| script: | | ||
| python3 -m venv .venv | ||
| source .venv/bin/activate | ||
| python3 -m pip install --upgrade pip | ||
| pip install semgrep | ||
| if [ $(Build.SourceBranchName) = "master" ]; then | ||
| export SEMGREP_BRANCH=$(Build.SourceBranchName) | ||
| echo "Semgrep full scan of master" | ||
| semgrep ci | ||
| elif [ $(System.PullRequest.PullRequestId) -ge 0 ]; then | ||
| echo "Semgrep diff scan" | ||
| git fetch origin master:origin/master | ||
| export SEMGREP_PR_ID=$(System.PullRequest.PullRequestId) | ||
| export SEMGREP_BASELINE_REF='origin/master' | ||
| semgrep ci | ||
| fi | ||
| ``` | ||
|
|
||
| :::info Customizing the configuration | ||
| * If your self-hosted runner [agent pool](https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/pools-queues?view=azure-devops&tabs=yaml%2Cbrowser) has a different name, update the `name` key under `pool` to match the desired agent pool. | ||
| * If your default branch is not called `master`, update the references to `master` to match the name of your default branch. | ||
| ::: | ||
|
|
||
| <AzureVariables /> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,4 @@ | ||
| ```yaml | ||
| variables: | ||
| - group: Semgrep_Variables | ||
|
|
||
| steps: | ||
| - checkout: self | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| ### Set environment variables in Azure Pipelines | ||
|
|
||
| Semgrep minimally requires the variable `SEMGREP_APP_TOKEN` in order to report results to the platform, and other variables may be helpful as well. To set these variables in Azure Pipelines: | ||
|
|
||
| 1. Set up a [variable group](https://learn.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups?view=azure-devops&tabs=classic) called `Semgrep_Variables`. | ||
| 2. Set `SEMGREP_APP_TOKEN` in the variable group, following the steps for [secret variables](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/set-secret-variables?view=azure-devops&tabs=yaml%2Cbash#set-a-secret-variable-in-a-variable-group). The variable is mapped into the `env` in the provided config. | ||
| 3. Optional: Add the following environment variables to the group if you aren't seeing hyperlinks to the code that generated a finding, or if you are not receiving PR or MR comments. Review the use of these variables at [Environment variables for creating hyperlinks in Semgrep AppSec Platform](https://semgrep.dev/docs/semgrep-ci/ci-environment-variables#environment-variables-for-creating-hyperlinks-in-semgrep-appsec-platform).These variables are not sensitive and do not need to be secret variables. | ||
| * `SEMGREP_REPO_NAME` | ||
| * `SEMGREP_REPO_URL` | ||
| * `SEMGREP_BRANCH` | ||
| * `SEMGREP_COMMIT` | ||
| * `SEMGREP_JOB_URL` | ||
| 4. Set variables for diff-aware scanning. The provided config sets `SEMGREP_PR_ID` to the system variable `System.PullRequest.PullRequestId` and `SEMGREP_BASELINE_REF` to `origin/master` within the `script` section of the config. The value of `SEMGREP_BASELINE_REF` is typically your trunk or default branch, so if you use a different branch than master, update the name accordingly. as `main` or `master`. | ||
| * If you prefer not to implement diff-aware scanning, you can skip setting these variables and remove the `elif` section of the `script` step. |