Skip to content

Commit e3a7d4d

Browse files
CeredronCeredron
Ceredron
and
Ceredron
committed
Setup our own Github runners (#1887)
* Setup our own Github runners * Run all workflows except for use case using the new runners * Coderabbit suggested fixes * Simplified labels * Hard-coded more and modularized to support multi-repo * Fixed validation error to be more user-friendly * Simplified even more * Hardcoded to use test environment * Install az cli and docker in our runners * Added doc on USE_SELF_HOSTED_RUNNERS * Vertical scaling * Consumption plan * Add missing party urn call * Fixes from coderabbit --------- Co-authored-by: Ceredron <roar.mjelde@digdir.no>
1 parent c340875 commit e3a7d4d

19 files changed

Lines changed: 528 additions & 30 deletions

‎.azure/applications/githubRunner/Dockerfile‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
FROM ubuntu:24.04
2+
ARG RUNNER_VERSION=2.334.0
3+
ARG TARGETARCH=amd64
4+
ENV DEBIAN_FRONTEND=noninteractive
5+
# Remove RUNNER_ALLOW_RUNASROOT — it's only needed when running as root, which you're not
6+
7+
RUN apt-get update \
8+
&& apt-get install -y --no-install-recommends \
9+
apt-transport-https \
10+
ca-certificates \
11+
curl \
12+
git \
13+
gnupg \
14+
jq \
15+
libicu-dev \
16+
lsb-release \
17+
tar \
18+
unzip \
19+
wget \
20+
&& mkdir -p /etc/apt/keyrings \
21+
&& curl -sL https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | tee /etc/apt/keyrings/microsoft.gpg > /dev/null \
22+
&& SUITE=$(lsb_release -cs) \
23+
&& echo "deb [arch=${TARGETARCH} signed-by=/etc/apt/keyrings/microsoft.gpg] https://packages.microsoft.com/repos/azure-cli/ $SUITE main" > /etc/apt/sources.list.d/azure-cli.list \
24+
&& apt-get update \
25+
&& apt-get install -y --no-install-recommends azure-cli \
26+
&& az aks install-cli --install-location /usr/local/bin/kubectl --kubelogin-install-location /usr/local/bin/kubelogin \
27+
&& rm -rf /var/lib/apt/lists/*
28+
RUN useradd -m -d /home/runner -s /bin/bash runner
29+
WORKDIR /home/runner
30+
RUN case "${TARGETARCH}" in \
31+
amd64) RUNNER_ARCH="linux-x64" ;; \
32+
arm64) RUNNER_ARCH="linux-arm64" ;; \
33+
arm) RUNNER_ARCH="linux-arm" ;; \
34+
*) echo "Unsupported TARGETARCH: ${TARGETARCH}" && exit 1 ;; \
35+
esac \
36+
&& ARCHIVE_NAME="actions-runner-${RUNNER_ARCH}-${RUNNER_VERSION}.tar.gz" \
37+
&& curl -fsSL -o "${ARCHIVE_NAME}" "https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/${ARCHIVE_NAME}" \
38+
&& tar xzf "${ARCHIVE_NAME}" \
39+
&& rm "${ARCHIVE_NAME}" \
40+
&& ./bin/installdependencies.sh
41+
COPY entrypoint.sh /entrypoint.sh
42+
RUN chmod +x /entrypoint.sh && chown runner:runner /entrypoint.sh && chown -R runner:runner /home/runner
43+
USER runner
44+
ENTRYPOINT ["/entrypoint.sh"]

‎.azure/applications/githubRunner/README.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# GitHub Runner Image
2+
3+
This folder contains a container image for self-hosted GitHub Actions runners in Azure Container Apps.
4+
5+
The image is built on Ubuntu and installs the official GitHub Actions runner binaries, Docker, Azure CLI, kubectl, and kubelogin.
6+
At startup, it:
7+
8+
1. Requests a temporary registration token from GitHub using `GITHUB_TOKEN`.
9+
2. Registers itself as an ephemeral runner for one repository (`GITHUB_URL`).
10+
3. Executes one job and then exits.
11+
4. Removes its runner registration on shutdown.
12+
5. Starts a Docker daemon so workflows can run `docker build/push` and `az` commands.
13+
14+
## Required GitHub Repository Secrets
15+
16+
The `manage-github-runners.yml` workflow expects these repository secrets:
17+
18+
- `AZURE_CLIENT_ID`
19+
- `AZURE_TENANT_ID`
20+
- `AZURE_SUBSCRIPTION_ID`
21+
- `AZURE_NAME_PREFIX`
22+
- `AZURE_ENVIRONMENT`
23+
- `AZURE_ENVIRONMENT_KEY_VAULT_NAME`
24+
25+
The Azure Key Vault referenced by `AZURE_ENVIRONMENT_KEY_VAULT_NAME` must also contain:
26+
27+
- `github-runner-token` (GitHub PAT/app token with permissions to manage self-hosted runners)
28+
29+
## Required GitHub Repository Variable
30+
31+
Set repository variable `USE_SELF_HOSTED_RUNNERS` to control workflow runner selection:
32+
33+
- `true`: workflows using the conditional `runs-on` expression run on `self-hosted`
34+
- any other value (or unset): workflows fall back to `ubuntu-latest`
35+
36+
The `manage-github-runners.yml` workflow controls the runner infrastructure itself, while this variable controls whether eligible workflows actually target self-hosted runners.

‎.azure/applications/githubRunner/entrypoint.sh‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
4+
if [[ -z "${GITHUB_URL:-}" ]]; then
5+
echo "GITHUB_URL is required."
6+
exit 1
7+
fi
8+
9+
if [[ -z "${GITHUB_TOKEN:-}" ]]; then
10+
echo "GITHUB_TOKEN is required."
11+
exit 1
12+
fi
13+
14+
if [[ "${GITHUB_URL}" != https://github.com/* ]]; then
15+
echo "GITHUB_URL must start with https://github.com/."
16+
exit 1
17+
fi
18+
19+
repo_path="${GITHUB_URL#https://github.com/}"
20+
repo_path="${repo_path%/}"
21+
IFS='/' read -r -a parts <<< "${repo_path}"
22+
owner=""
23+
repo=""
24+
if [[ ${#parts[@]} -eq 2 ]]; then
25+
owner="${parts[0]}"
26+
repo="${parts[1]}"
27+
fi
28+
29+
if [[ ${#parts[@]} -ne 2 || -z "${owner}" || -z "${repo}" ]]; then
30+
echo "GITHUB_URL must be in the format https://github.com/<owner>/<repo>."
31+
exit 1
32+
fi
33+
34+
short_host="$(hostname | cut -c1-32)"
35+
runner_name="runner-${short_host}-${RANDOM}"
36+
runner_name="${runner_name:0:64}"
37+
work_folder="_work"
38+
DOCKERD_PID=""
39+
40+
api_url="https://api.github.com/repos/${owner}/${repo}/actions/runners"
41+
common_headers=(
42+
-H "Accept: application/vnd.github+json"
43+
-H "Authorization: Bearer ${GITHUB_TOKEN}"
44+
-H "X-GitHub-Api-Version: 2022-11-28"
45+
)
46+
47+
echo "Requesting registration token for ${owner}/${repo}..."
48+
registration_token="$(
49+
curl -fsSL -X POST "${common_headers[@]}" "${api_url}/registration-token" | jq -r '.token'
50+
)"
51+
52+
if [[ -z "${registration_token}" || "${registration_token}" == "null" ]]; then
53+
echo "Failed to fetch runner registration token."
54+
exit 1
55+
fi
56+
57+
remove_runner() {
58+
set +e
59+
echo "Removing runner registration..."
60+
remove_token="$(curl -fsSL -X POST "${common_headers[@]}" "${api_url}/remove-token" | jq -r '.token')"
61+
if [[ -n "${remove_token}" && "${remove_token}" != "null" ]]; then
62+
./config.sh remove --unattended --token "${remove_token}" >/dev/null 2>&1
63+
fi
64+
}
65+
66+
trap remove_runner EXIT INT TERM
67+
68+
./config.sh \
69+
--unattended \
70+
--replace \
71+
--ephemeral \
72+
--name "${runner_name}" \
73+
--work "${work_folder}" \
74+
--url "${GITHUB_URL}" \
75+
--token "${registration_token}"
76+
77+
echo "Runner ${runner_name} configured. Waiting for a single job..."
78+
./run.sh --disableupdate

‎.azure/applications/githubRunner/main.bicep‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
targetScope = 'subscription'
2+
3+
@minLength(3)
4+
param location string
5+
@secure()
6+
@minLength(3)
7+
param sourceKeyVaultName string
8+
@secure()
9+
param keyVaultUrl string
10+
@secure()
11+
param namePrefix string
12+
@description('GitHub registration URL, e.g. https://github.com/Altinn/altinn-correspondence')
13+
param githubUrl string
14+
@description('Container image for the self-hosted runner.')
15+
param runnerImage string = 'ghcr.io/altinn/altinn-correspondence-github-runner:latest'
16+
17+
var resourceGroupName = '${namePrefix}-rg'
18+
var appName = '${namePrefix}-github-runner'
19+
20+
resource resourceGroup 'Microsoft.Resources/resourceGroups@2024-11-01' existing = {
21+
name: resourceGroupName
22+
}
23+
24+
resource keyvault 'Microsoft.KeyVault/vaults@2024-11-01' existing = {
25+
name: sourceKeyVaultName
26+
scope: resourceGroup
27+
}
28+
29+
module runnerIdentity '../../modules/identity/create.bicep' = {
30+
name: 'githubRunnerIdentity'
31+
scope: resourceGroup
32+
params: {
33+
namePrefix: '${namePrefix}-github-runner'
34+
location: location
35+
}
36+
}
37+
38+
module keyvaultAddReaderRolesRunnerIdentity '../../modules/keyvault/addReaderRoles.bicep' = {
39+
name: 'kvreader-${namePrefix}-github-runner'
40+
scope: resourceGroup
41+
params: {
42+
keyvaultName: sourceKeyVaultName
43+
principals: [
44+
{ objectId: runnerIdentity.outputs.principalId, principalType: 'ServicePrincipal' }
45+
]
46+
}
47+
}
48+
49+
module githubRunnerContainerApp '../../modules/githubRunnerContainerApp/main.bicep' = {
50+
name: appName
51+
scope: resourceGroup
52+
dependsOn: [
53+
keyvaultAddReaderRolesRunnerIdentity
54+
]
55+
params: {
56+
namePrefix: namePrefix
57+
location: location
58+
userAssignedIdentityResourceId: runnerIdentity.outputs.id
59+
keyVaultUrl: keyVaultUrl
60+
containerAppEnvId: keyvault.getSecret('container-app-env-id')
61+
runnerImage: runnerImage
62+
githubUrl: githubUrl
63+
}
64+
}
65+
66+
output name string = githubRunnerContainerApp.outputs.name
67+
output revisionName string = githubRunnerContainerApp.outputs.revisionName

‎.azure/applications/githubRunner/params.bicepparam‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
using './main.bicep'
2+
3+
param namePrefix = readEnvironmentVariable('NAME_PREFIX')
4+
param location = 'norwayeast'
5+
6+
// secrets
7+
param sourceKeyVaultName = readEnvironmentVariable('KEY_VAULT_NAME')
8+
param keyVaultUrl = readEnvironmentVariable('KEY_VAULT_URL')
9+
10+
// GitHub runner settings
11+
param githubUrl = readEnvironmentVariable('GITHUB_URL')
12+
param runnerImage = readEnvironmentVariable('GITHUB_RUNNER_IMAGE', 'ghcr.io/altinn/altinn-correspondence-github-runner:latest')

‎.azure/modules/githubRunnerContainerApp/main.bicep‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
param location string
2+
@secure()
3+
param namePrefix string
4+
@secure()
5+
param userAssignedIdentityResourceId string
6+
@secure()
7+
param keyVaultUrl string
8+
@secure()
9+
param containerAppEnvId string
10+
@description('Container image for the self-hosted runner.')
11+
param runnerImage string
12+
@description('GitHub registration URL, e.g. https://github.com/Altinn/altinn-correspondence')
13+
param githubUrl string
14+
15+
var containerAppName = '${namePrefix}-github-runner'
16+
var githubTokenSecretRefName = 'github-runner-token'
17+
var githubTokenSecretName = 'github-runner-token'
18+
var targetQueueLength = 1
19+
var cooldownPeriodSeconds = 3600
20+
var pollingIntervalSeconds = 30
21+
var maxReplicas = 4
22+
var containerAppResources = {
23+
cpu: 2
24+
memory: '4.0Gi'
25+
}
26+
27+
var secrets = [
28+
{
29+
identity: userAssignedIdentityResourceId
30+
keyVaultUrl: '${keyVaultUrl}/secrets/${githubTokenSecretName}'
31+
name: githubTokenSecretRefName
32+
}
33+
]
34+
35+
var containerAppEnvVars = [
36+
{ name: 'RUNNER_SCOPE', value: 'repo' }
37+
{ name: 'GITHUB_URL', value: githubUrl }
38+
{ name: 'DISABLE_AUTO_UPDATE', value: 'true' }
39+
]
40+
41+
resource githubRunnerContainerApp 'Microsoft.App/containerApps@2026-01-01' = {
42+
name: containerAppName
43+
location: location
44+
tags: resourceGroup().tags
45+
identity: {
46+
type: 'UserAssigned'
47+
userAssignedIdentities: {
48+
'${userAssignedIdentityResourceId}': {}
49+
}
50+
}
51+
properties: {
52+
environmentId: containerAppEnvId
53+
configuration: {
54+
activeRevisionsMode: 'Single'
55+
secrets: secrets
56+
}
57+
template: {
58+
scale: {
59+
minReplicas: 0
60+
maxReplicas: maxReplicas
61+
cooldownPeriod: cooldownPeriodSeconds
62+
pollingInterval: pollingIntervalSeconds
63+
rules: [
64+
{
65+
name: 'github-runner-queue'
66+
custom: {
67+
type: 'github-runner'
68+
metadata: {
69+
githubApiURL: 'https://api.github.com'
70+
owner: split(replace(githubUrl, 'https://github.com/', ''), '/')[0]
71+
repos: split(replace(githubUrl, 'https://github.com/', ''), '/')[1]
72+
targetWorkflowQueueLength: string(targetQueueLength)
73+
runnerScope: 'repo'
74+
}
75+
auth: [
76+
{
77+
secretRef: githubTokenSecretRefName
78+
triggerParameter: 'personalAccessToken'
79+
}
80+
]
81+
}
82+
}
83+
]
84+
}
85+
containers: [
86+
{
87+
name: 'github-runner'
88+
image: runnerImage
89+
env: concat(containerAppEnvVars, [
90+
{
91+
name: 'GITHUB_TOKEN'
92+
secretRef: githubTokenSecretRefName
93+
}
94+
])
95+
resources: containerAppResources
96+
}
97+
]
98+
}
99+
}
100+
}
101+
102+
output name string = githubRunnerContainerApp.name
103+
output revisionName string = githubRunnerContainerApp.properties.latestRevisionName
104+
output app object = githubRunnerContainerApp

‎.github/workflows/check-label-for-pr.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ on:
44
types: [opened, reopened, ready_for_review, labeled, unlabeled]
55
jobs:
66
label:
7-
runs-on: ubuntu-latest
7+
runs-on: ${{ vars.USE_SELF_HOSTED_RUNNERS == 'true' && 'self-hosted' || 'ubuntu-latest' }}
88
permissions:
99
pull-requests: write
1010
steps:

0 commit comments

Comments
 (0)