Skip to content

Commit b4df7da

Browse files
caneraltinbasakcaneraltinbasak
committed
feat: Add meta-chromium-test layer with CI/CD infrastructure
This commit introduces a complete testing ecosystem for Chromium and Electron builds on embedded Linux systems, featuring a new Yocto/OpenEmbedded layer (meta-chromium-test) and AWS-based CI/CD infrastructure that supports automated builds, testing, and validation across multiple architectures and display backends. The new meta-chromium-test layer provides a full Yocto/OpenEmbedded testing infrastructure with proper layer structure, supporting ARM, AArch64, RISC-V, and x86-64 architectures across multiple display backends including Ozone Wayland, X11, and Ozone X11. The layer uses KAS for reproducible build system configuration management and automatically adapts to the current meta-browser branch, whether it's scarthgap, master, or other versions. This eliminates the need to maintain separate configurations for each Yocto version. The testing components include smoke test suites for automated Chromium and Electron functionality validation, QEMU integration for emulated testing environments across all architectures, and BitBake recipes for chromium-tests and electron-tests packages. The layer includes validation scripts that provide a framework for build and test validation, ensuring reliability across the entire testing pipeline. The AWS-based CI/CD infrastructure provides scalable build capabilities through EC2 auto-scaling that dynamically provisions build runners based on workflow demand. The system includes automatic resource cleanup preventing orphaned infrastructure costs, EFS shared caching for distributed sstate-cache and downloads across all build nodes, and S3 artifact storage for persistent storage of build artifacts and test results. This infrastructure supports matrix-based parallel builds covering all combinations of browser, architecture, and display backend, with each matrix combination running in an isolated environment where individual failures don't block other combinations. The workflow architecture consists of five GitHub Actions workflows that orchestrate the complete build and test pipeline. The main build/test workflow includes an 11-hour timeout to accommodate long Yocto builds, while dedicated EC2 infrastructure management handles resource provisioning. Browser-specific workflows for Chromium and Electron provide targeted testing, and automated EFS cache cleanup runs weekly to prevent cache bloat. The system supports both AWS-based auto-scaling and self-hosted runners, providing flexibility when AWS infrastructure is unavailable. Build optimization features include shared sstate-cache that reduces build times through artifact reuse, persistent downloads caching for fetched sources across builds, and automated weekly cleanup preventing cache bloat. The multi-node scaling capability distributes builds across multiple EC2 instances, while automatic provisioning creates EC2 instances on-demand for build jobs. Resources are automatically terminated after builds complete, implementing a pay-per-use model with automatic resource lifecycle management. The testing validation system includes built-in smoke tests that verify basic Chromium and Electron functionality, QEMU emulation that runs tests in environments matching target architectures, and build validation checks ensuring successful compilation and packaging. Detailed failure analysis and artifact collection provide error reporting when issues occur. This testing matrix supports all combinations of three architectures (ARM, AArch64, x86-64) with both Chromium and Electron browsers across multiple display backends. Chromium supports Ozone Wayland and X11, while Electron supports Ozone Wayland and Ozone X11, all built against glibc (musl support was removed for simplicity). The KAS file generation system uses script-based creation of configuration files with a template system ensuring consistent configuration across all combinations, making it easy to add new architectures or backends while maintaining efficiency through a single source of truth for configuration patterns.
1 parent 198b0f7 commit b4df7da

44 files changed

Lines changed: 2780 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/bs_meta_browser_build_and_test.yml‎

Lines changed: 451 additions & 0 deletions
Large diffs are not rendered by default.

‎.github/workflows/bs_meta_browser_ci_ec2.yml‎

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
name: 'Build and Test Meta-Browser: EC2 controller'
2+
on:
3+
workflow_call:
4+
inputs:
5+
build_type:
6+
description: 'Build Type'
7+
required: true
8+
type: string
9+
10+
browser:
11+
description: 'Chromium or Electron'
12+
required: true
13+
type: string
14+
15+
chromium_version:
16+
description: 'Display backend/Ozone platform (ozone-wayland or x11)'
17+
required: true
18+
type: string
19+
20+
arch:
21+
description: 'Target architecture'
22+
required: true
23+
type: string
24+
25+
instance_type:
26+
description: 'EC2 instance type'
27+
required: false
28+
type: string
29+
default: c6a.4xlarge
30+
31+
leave_ec2_instance_running:
32+
description: 'Leave EC2 instance running after use'
33+
type: boolean
34+
default: false
35+
36+
instance_name_postfix:
37+
description: 'Name to add as postfix to the EC2 machine'
38+
type: string
39+
default: auto-triggered
40+
41+
aws_arn_role:
42+
required: true
43+
type: string
44+
45+
aws_region:
46+
required: true
47+
type: string
48+
49+
jobs:
50+
start-runner:
51+
name: Start self-hosted EC2 runner
52+
runs-on: ubuntu-latest
53+
outputs:
54+
label: ${{ steps.start-ec2-runner.outputs.label }}
55+
ec2-instance-id: ${{ steps.start-ec2-runner.outputs.ec2-instance-id }}
56+
steps:
57+
- name: Configure AWS credentials
58+
uses: aws-actions/configure-aws-credentials@v4
59+
with:
60+
aws-region: ${{ inputs.aws_region }}
61+
role-to-assume: ${{ inputs.aws_arn_role }}
62+
role-session-name: mb-ci-${{ github.run_id }}
63+
role-duration-seconds: 43200 # 12 hours for long builds
64+
65+
- name: Download config file and set env vars from it
66+
run: |
67+
aws s3 cp s3://meta-browser-ci-config-bucket/config.json .
68+
aws s3 cp s3://meta-browser-ci-config-bucket/set_github_env_vars.py .
69+
python3 set_github_env_vars.py --file config.json
70+
71+
- name: Clean up any leftover runners
72+
run: |
73+
echo "Checking for any leftover runners from previous runs..."
74+
echo "Current GitHub run context:"
75+
echo "Run ID: ${{ github.run_id }}"
76+
echo "Run attempt: ${{ github.run_attempt }}"
77+
echo "Run number: ${{ github.run_number }}"
78+
echo "This run will use unique label with attempt number to avoid conflicts"
79+
80+
- name: Start EC2 runner with retry
81+
id: start-ec2-runner
82+
uses: brightsign/ec2-github-runner@0fa8b183dd4124fd191ccdbc48b68f0ea46a9634
83+
timeout-minutes: 15 # Allow more time for registration
84+
with:
85+
mode: start
86+
github-app-private-key: ${{ secrets.GH_APP_PRIVATE_KEY }}
87+
github-app-id: 287690
88+
ec2-image-id: ami-08a4255385679596c # Custom AMI with Yocto build dependencies pre-installed
89+
ec2-instance-type: ${{ inputs.instance_type }}
90+
subnet-id: ${{ env.VPC_SUBNET_ID }}
91+
security-group-id: ${{ env.VPC_SG_ID }}
92+
run-as-service-with-user: ubuntu
93+
label: "mb-${{ github.run_id }}-${{ github.run_attempt }}-${{ inputs.browser }}-${{ inputs.arch }}-${{ inputs.chromium_version }}" # Unique per matrix job with attempt
94+
aws-resource-tags: > # optional, requires additional permissions
95+
[
96+
{"Key": "Name", "Value": "github-runner-meta-browser-${{ inputs.instance_name_postfix }}-${{ github.run_id }}"},
97+
{"Key": "GitHubRepository", "Value": "${{ github.repository }}"},
98+
{"Key": "ChromiumVersion", "Value": "${{ inputs.chromium_version }}"},
99+
{"Key": "Architecture", "Value": "${{ inputs.arch }}"},
100+
{"Key": "RunId", "Value": "${{ github.run_id }}"},
101+
{"Key": "RunAttempt", "Value": "${{ github.run_attempt }}"},
102+
{"Key": "Browser", "Value": "${{ inputs.browser }}"}
103+
]
104+
105+
build-and-test-meta-browser:
106+
name: Build and Test Meta-Browser
107+
needs: start-runner # required to start the main job when the runner is ready
108+
uses: ./.github/workflows/bs_meta_browser_build_and_test.yml
109+
secrets: inherit
110+
with:
111+
runner_name: ${{ needs.start-runner.outputs.label }} # run the job on the newly created runner
112+
github_hosted_runner: false
113+
browser: ${{ inputs.browser }}
114+
build_type: ${{ inputs.build_type }}
115+
chromium_version: ${{ inputs.chromium_version }}
116+
arch: ${{ inputs.arch }}
117+
aws_arn_role: ${{ inputs.aws_arn_role }}
118+
aws_region: ${{ inputs.aws_region }}
119+
120+
stop-runner:
121+
name: Stop self-hosted EC2 runner
122+
needs:
123+
- start-runner # required to get output from the start-runner job
124+
- build-and-test-meta-browser # required to wait when the main job is done
125+
runs-on: ubuntu-latest
126+
if: ${{ always() }} # required to stop the runner even if the error happened in the previous jobs
127+
steps:
128+
- name: Configure AWS credentials
129+
uses: aws-actions/configure-aws-credentials@v4
130+
with:
131+
role-to-assume: ${{ inputs.aws_arn_role }}
132+
aws-region: ${{ inputs.aws_region }}
133+
role-session-name: mb-cleanup-${{ github.run_id }}
134+
135+
- name: Stop EC2 runner
136+
uses: brightsign/ec2-github-runner@0fa8b183dd4124fd191ccdbc48b68f0ea46a9634
137+
with:
138+
mode: stop
139+
github-app-private-key: ${{ secrets.GH_APP_PRIVATE_KEY }}
140+
github-app-id: 287690
141+
label: ${{ needs.start-runner.outputs.label }}
142+
ec2-instance-id: ${{ needs.start-runner.outputs.ec2-instance-id }}
143+
leave-ec2-instance-running: ${{ inputs.leave_ec2_instance_running }}

‎.github/workflows/chromium.yml‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
name: Chromium build- and smoke-test
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
repository:
7+
description: 'Repository to clone for the workflow'
8+
required: true
9+
default: 'OSSystems'
10+
branch:
11+
description: 'Branch to checkout for the workflow'
12+
required: true
13+
default: 'master'
14+
use_aws:
15+
description: 'Use AWS EC2 instances instead of local runner'
16+
required: false
17+
type: boolean
18+
default: true
19+
instance_type:
20+
description: 'EC2 instance type for build (more cores = faster builds)'
21+
required: false
22+
type: choice
23+
options:
24+
- 'c6id.4xlarge' # 16 vCPUs, 32 GB RAM, 1x 950 GB NVMe
25+
- 'c6id.8xlarge' # 32 vCPUs, 64 GB RAM, 1x 1900 GB NVMe
26+
- 'c6id.12xlarge' # 48 vCPUs, 96 GB RAM, 2x 1425 GB NVMe
27+
- 'c6id.16xlarge' # 64 vCPUs, 128 GB RAM, 2x 1900 GB NVMe
28+
default: 'c6id.8xlarge'
29+
pull_request:
30+
branches:
31+
- master
32+
paths:
33+
- 'meta-chromium/recipes-browser/chromium/files/**'
34+
- 'meta-chromium/recipes-browser/chromium/chromium*'
35+
- 'meta-chromium/recipes-browser/chromium/gn*'
36+
- '.github/workflows/chromium.yml'
37+
38+
permissions:
39+
contents: read
40+
actions: read
41+
checks: write
42+
id-token: write # Required for OIDC authentication
43+
44+
jobs:
45+
# AWS-based builds (always for PR, default for manual dispatch)
46+
aws-matrix-build:
47+
if: ${{ (github.repository_owner == 'brightsign' || github.repository_owner == 'OSSystems') && (github.event_name == 'pull_request' || inputs.use_aws == true || inputs.use_aws == null) }}
48+
strategy:
49+
fail-fast: false # Continue other matrix jobs even if one fails
50+
matrix:
51+
ozone_platform: [ozone-wayland, x11]
52+
arch: [arm, aarch64, x86-64]
53+
uses: ./.github/workflows/bs_meta_browser_ci_ec2.yml
54+
secrets: inherit
55+
with:
56+
build_type: "release"
57+
browser: "chromium"
58+
chromium_version: ${{ matrix.ozone_platform }}
59+
arch: ${{ matrix.arch }}
60+
aws_arn_role: "arn:aws:iam::195607249165:role/github-actions-meta-browser-repo"
61+
aws_region: "us-east-1"
62+
instance_type: ${{ inputs.instance_type || 'c6id.8xlarge' }} # Default for PR builds, user choice for manual
63+
64+
# Local runner (manual dispatch only, when explicitly disabled AWS)
65+
local-build:
66+
if: ${{ (github.repository_owner == 'brightsign' || github.repository_owner == 'OSSystems') && github.event_name == 'workflow_dispatch' && inputs.use_aws == false }}
67+
strategy:
68+
fail-fast: false # Continue other matrix jobs even if one fails
69+
matrix:
70+
browser_version: [ozone-wayland, x11]
71+
browser: [chromium]
72+
arch: [arm, aarch64, x86-64]
73+
runs-on: [self-hosted, chromium]
74+
container:
75+
image: skandigraun/yocto:latest
76+
volumes:
77+
- yocto:/yocto
78+
steps:
79+
- run: |
80+
mkdir -p /yocto
81+
cd /yocto
82+
rm -rf meta-browser
83+
# Clean stale pseudo state from any previous interrupted builds
84+
rm -rf build/tmp/work/*/*/*/pseudo build/tmp/sysroots-components/*/pseudo 2>/dev/null || true
85+
if [ "${{ github.event_name }}" = "pull_request" ]; then
86+
GH_URL="$GITHUB_SERVER_URL/${{ github.event.pull_request.head.repo.full_name }}"
87+
GH_REV="$GITHUB_HEAD_REF"
88+
else
89+
GH_URL="$GITHUB_SERVER_URL/${{ github.repository }}"
90+
GH_REV="${{ github.ref_name }}"
91+
fi
92+
git clone $GH_URL
93+
git -C meta-browser checkout $GH_REV
94+
# meta-chromium-test is now integrated into meta-browser
95+
./meta-browser/meta-chromium-test/scripts/build.sh ${{ matrix.arch }} ${{ matrix.browser_version }} ${{ matrix.browser }}

0 commit comments

Comments
 (0)