Skip to content

Add Multi-Board Treadmill Descriptors & BLE Advertise+Scan Test #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 31 commits into from
Closed
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
7af45c3
Multi-Board Treadmill Descriptors & BLE Advertise+Scan Test init
charles37 Jan 31, 2025
de736f2
Enhanced BLE Advertising Scanning Test
charles37 Mar 11, 2025
54a7f6a
ble workflow
charles37 Mar 20, 2025
36b4627
Merge branch 'main' into dev/mult-board-treadmill
charles37 Mar 20, 2025
d27c661
change workflow to trigger on commit to dev/multi-board-treadmill
charles37 Mar 20, 2025
9a798a4
Merge branch 'dev/mult-board-treadmill' of https://github.com/tock/to…
charles37 Mar 20, 2025
f89516a
change workflow for board path modifications
charles37 Mar 20, 2025
07168d3
modify start path for testing script
charles37 Mar 20, 2025
3babc3d
test commit
charles37 Mar 20, 2025
5b4cd0b
change workflow for board path modifications
charles37 Mar 21, 2025
ceae7a8
test commit with ls for debug
charles37 Mar 21, 2025
43abda4
test
charles37 Mar 21, 2025
63e17f4
test commit
charles37 Mar 24, 2025
0bb9b25
test commit
charles37 Mar 25, 2025
93bdd4f
test commit
charles37 Mar 25, 2025
a57f4ba
correct paths
charles37 Mar 25, 2025
9dfbb12
test
charles37 Mar 25, 2025
c4988b3
test
charles37 Mar 27, 2025
be1c21f
test commit
charles37 Mar 27, 2025
8d91f3c
test commit for logging
charles37 Mar 27, 2025
90492c4
finally fixed
charles37 Mar 27, 2025
5364663
fix faulty condition
charles37 Mar 27, 2025
46223df
test commit
charles37 Mar 31, 2025
726ed35
test commit
charles37 Apr 1, 2025
f1e46e7
test commit final
charles37 Apr 1, 2025
72863ad
fix test
charles37 Apr 17, 2025
1f90a9a
finally fixed this test
charles37 Apr 22, 2025
2614b9d
test
charles37 Apr 22, 2025
1d6e2b5
new manufacturer specific header check
charles37 Apr 24, 2025
b044856
remove manual tockloader install
charles37 Apr 24, 2025
022df4f
add tockloader back to requirements.txt
charles37 Apr 24, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 73 additions & 0 deletions .github/workflows/treadmill-ci-ble-test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Licensed under the Apache License, Version 2.0 or the MIT License.
# SPDX-License-Identifier: Apache-2.0 OR MIT
# Copyright Tock Contributors 2024.
# This workflow contains Treadmill-based hardware CI for nightly BLE testing.
#
# Treadmill is a distributed hardware testbed developed within the Tock OS
# project. For more information on Treadmill, have a look at its documentation
# [1] or repository [2].
#
# This workflow specifically targets BLE advertising and scanning tests on multiple
# boards attached to a specific supervisor.
#
# [1]: https://book.treadmill.ci/
# [2]: https://github.com/treadmill-tb/treadmill
# [3]: https://book.treadmill.ci/user-guide/github-actions-integration.html
# TEST
name: treadmill-ci-ble-test
env:
TERM: xterm # Makes tput work in actions output
# Controls when the action will run.
on:
# Manual trigger
workflow_dispatch:
inputs:
tock-kernel-ref:
description: 'Ref (revision/branch/tag) of the upstream Tock repo to test'
required: true
default: 'master'
libtock-c-ref:
description: 'Ref (revision/branch/tag) of the upstream libtock-c repo to test'
required: true
default: 'master'
# Add push trigger for your branch
push:
branches:
- dev/mult-board-treadmill
permissions:
contents: read
jobs:
prepare-ble-test:
runs-on: ubuntu-latest
outputs:
# This is a fixed list containing just the BLE advertising and scanning test
hwci-tests-json: ${{ steps.prepare-test.outputs.hwci-tests-json }}
steps:
- name: Prepare BLE test
id: prepare-test
run: |
# Instead of analyzing changes, we specifically select the BLE test
echo 'hwci-tests-json=["tests/ble_advertising_scanning_test.py"]' >> "$GITHUB_OUTPUT"
echo "Selected test: tests/ble_advertising_scanning_test.py"
run-treadmill-ci:
needs: [prepare-ble-test]
uses: ./.github/workflows/treadmill-ci.yml
with:
# Only run on a specific repository
repository-filter: 'tock/tock-hardware-ci'
# Provide access to the required Treadmill secrets
job-environment: 'treadmill-ci'
# This workflow tests the tock-hardware-ci scripts itself, so take the
# current GITHUB_SHA:
tock-hardware-ci-ref: ${{ github.sha }}
# Use the provided upstream Tock kernel / userspace components:
tock-kernel-ref: ${{ inputs.tock-kernel-ref }}
libtock-c-ref: ${{ inputs.libtock-c-ref }}
# Pass our fixed test JSON
tests-json: ${{ needs.prepare-ble-test.outputs.hwci-tests-json }}
# Specify that this is for the BLE test which needs multiple boards
multi-board: 'true'
supervisor-id: 'fb1384d5-e1a5-469c-beb4-0d4d215c9793'
board-descriptors: >-
board_descriptors/fb1384d5-e1a5-469c-beb4-0d4d215c9793/board-nrf52840dk-001050202501.yml board_descriptors/fb1384d5-e1a5-469c-beb4-0d4d215c9793/board-nrf52840dk-001050244773.yml
secrets: inherit
109 changes: 71 additions & 38 deletions .github/workflows/treadmill-ci.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
# Licensed under the Apache License, Version 2.0 or the MIT License.
# SPDX-License-Identifier: Apache-2.0 OR MIT
# Copyright Tock Contributors 2024.

# This workflow contains all Treadmill-based hardware CI jobs.
#
# Treadmill is a distributed hardware testbed developed within the Tock OS
Expand All @@ -17,12 +16,9 @@
# [2]: https://github.com/treadmill-tb/treadmill
# [3]: https://book.treadmill.ci/user-guide/github-actions-integration.html
#

name: treadmill-ci

env:
TERM: xterm # Makes tput work in actions output

on:
workflow_call:
inputs:
Expand All @@ -45,76 +41,78 @@ on:
required: false
type: string
default: '["tests/c_hello.py"]' # Default to single test for backward compatibility

# New inputs for multi-board tests
multi-board:
required: false
type: string
default: 'false'
supervisor-id:
required: false
type: string
default: ''
board-descriptors:
required: false
type: string
default: ''
Comment on lines +44 to +56
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add descriptions on what these inputs are for, and their expected format?

jobs:
test-prepare:
runs-on: ubuntu-latest

# Do not run job on forks, as they will not have the correct environment set up
if: github.repository == inputs.repository-filter

environment: ${{ inputs.job-environment }}

outputs:
tml-job-ids: ${{ steps.treadmill-job-launch.outputs.tml-job-ids }}
tml-jobs: ${{ steps.treadmill-job-launch.outputs.tml-jobs }}

steps:
- uses: actions-rust-lang/setup-rust-toolchain@v1

- name: Checkout Treadmill repository
uses: actions/checkout@v4
with:
repository: treadmill-tb/treadmill
# treadmill-tb/treadmill main as of Oct 1, 2024, 3:05 PM EDT
ref: 'c82f4d7ebddd17f8275ba52139e64e04623f30cb'
path: treadmill

- name: Cache Treadmill CLI compilation artifacts
id: cache-tml-cli
uses: actions/cache@v4
with:
path: treadmill/target
key: ${{ runner.os }}-tml-cli

- name: Compile the Treadmill CLI binary
run: |
pushd treadmill
cargo build --package tml-cli
popd
echo "$PWD/treadmill/target/debug" >> "$GITHUB_PATH"

- name: Generate a token to register new just-in-time runners
id: generate-token
uses: actions/create-github-app-token@v1
with:
app-id: ${{ vars.TREADMILL_GH_APP_CLIENT_ID }}
private-key: ${{ secrets.TREADMILL_GH_APP_PRIVATE_KEY }}

- name: Create GitHub just-in-time runners and enqueue Treadmill jobs
id: treadmill-job-launch
env:
GH_TOKEN: ${{ steps.generate-token.outputs.token }}
TML_API_TOKEN: ${{ secrets.TREADMILL_API_TOKEN }}

# Currently, all tests run only on hosts attached to an nRF52840DK
DUT_BOARD: nrf52840dk

# A Raspberry Pi OS netboot (NBD) image with a GitHub Actions
# self-hosted runner pre-configured.
#
# For the available images see
# https://book.treadmill.ci/treadmillci-deployment/images.html
IMAGE_ID: f94b8f8edd54321e6370d898f87ccbd2659a67ed0300fda2adc8099cdd157790

# Limit the supervisors to hosts that are compatible with this
# image. This is a hack until we introduce "image sets" which define
# multiple images for various supervisor hosts, but otherwise behave
# identically:
HOST_TYPE: nbd-netboot
HOST_ARCH: arm64

TESTS_JSON: ${{ inputs.tests-json }}
# Support for specific supervisor targeting for multi-board tests
MULTI_BOARD: ${{ inputs.multi-board }}
SUPERVISOR_ID: ${{ inputs.supervisor-id }}
run: |
# When we eventually launch tests on multiple hardware platforms in
# parallel, we need to supply different SUB_TEST_IDs here:
Expand Down Expand Up @@ -152,11 +150,22 @@ jobs:
}"

echo "Enqueueing treadmill job:"
TML_JOB_ID_JSON="$(tml job enqueue \
"$IMAGE_ID" \
--tag-config "board:$DUT_BOARD;host-type:$HOST_TYPE;host-arch:$HOST_ARCH" \
--parameters "$TML_JOB_PARAMETERS" \
)"

# If this is a multi-board test targeting a specific supervisor
if [ "$MULTI_BOARD" = "true" ] && [ -n "$SUPERVISOR_ID" ]; then
TML_JOB_ID_JSON="$(tml job enqueue \
"$IMAGE_ID" \
--tag-config "supervisor:$SUPERVISOR_ID;host-type:$HOST_TYPE;host-arch:$HOST_ARCH" \
--parameters "$TML_JOB_PARAMETERS" \
)"
else
# Default job enqueue for single board tests
TML_JOB_ID_JSON="$(tml job enqueue \
"$IMAGE_ID" \
--tag-config "board:$DUT_BOARD;host-type:$HOST_TYPE;host-arch:$HOST_ARCH" \
--parameters "$TML_JOB_PARAMETERS" \
)"
fi

TML_JOB_ID="$(echo "$TML_JOB_ID_JSON" | jq -r .job_id)"
echo "Enqueued Treadmill job with ID $TML_JOB_ID"
Expand All @@ -179,9 +188,12 @@ jobs:
|------|-------|-----|
GITHUB_STEP_SUMMARY
echo "$TESTS_JSON" | jq -r -c '.[]' | while read TEST; do
echo "| \`$TEST\` | \`$DUT_BOARD\` | [\`$TML_JOB_ID\`](#tml-job-summary-$TML_JOB_ID) |" >>"$GITHUB_STEP_SUMMARY"
BOARD_INFO="$DUT_BOARD"
if [ "$MULTI_BOARD" = "true" ]; then
BOARD_INFO="Multiple boards on supervisor $SUPERVISOR_ID"
fi
echo "| \`$TEST\` | \`$BOARD_INFO\` | [\`$TML_JOB_ID\`](#tml-job-summary-$TML_JOB_ID) |" >>"$GITHUB_STEP_SUMMARY"
done

test-execute:
needs: test-prepare
strategy:
Expand All @@ -192,7 +204,7 @@ jobs:
- name: Print Treadmill Job Context and Debug Information
run: |
echo "Treadmill job id: ${{ matrix.tml-job-id }}"
echo "GitHub Actions Runner ID: ${{ fromJSON(needs.test-prepare.outputs.tml-jobs)[matrix.tml-job-id] }}"
echo "GitHub Actions Runner ID: ${{ fromJSON(needs.test-prepare.outputs.tml-jobs)[matrix.tml-job-id].runner-id }}"
echo "Host ID: $(cat /run/tml/host-id)"
echo "===== Parameters: ====="
ls /run/tml/parameters
Expand All @@ -207,16 +219,13 @@ jobs:
ls -lh /dev/ttyACM* 2>/dev/null || true
ls -lh /dev/ttyUSB* 2>/dev/null || true
ls -lh /dev/bus/usb/*/* 2>/dev/null || true

- name: Disable wget progress output
run: |
echo "verbose = off" >> $HOME/.wgetrc

- uses: actions-rust-lang/setup-rust-toolchain@v1
with:
# Avoid overwriting the RUSTFLAGS environment variable
rustflags: ''

# This is required for the actions/checkout steps to perform a
# proper git clone that also supports checking out submodules:
- name: Install git
Expand All @@ -228,20 +237,17 @@ jobs:
# part out.
sudo DEBIAN_FRONTEND=noninteractive apt update || true
sudo DEBIAN_FRONTEND=noninteractive apt install -y git

- name: Checkout the Tock Hardware CI scripts
uses: actions/checkout@v4
with:
repository: tock/tock-hardware-ci
ref: ${{ inputs.tock-hardware-ci-ref }}

- name: Checkout the Tock kernel repository
uses: actions/checkout@v4
with:
path: hwci/repos/tock
repository: tock/tock
ref: ${{ inputs.tock-kernel-ref }}

- name: Checkout the libtock-c repository
uses: actions/checkout@v4
with:
Expand All @@ -251,24 +257,33 @@ jobs:
fetch-depth: 0
submodules: false
persist-credentials: true

- name: Run setup script
run: |
cd ./hwci/
./setup.sh

- name: Run tests
env:
JSON_TEST_ARRAY: ${{ toJSON(fromJSON(needs.test-prepare.outputs.tml-jobs)[matrix.tml-job-id].tests) }}
MULTI_BOARD: ${{ inputs.multi-board }}
SUPERVISOR_ID: ${{ inputs.supervisor-id }}
BOARD_DESCRIPTORS: ${{ inputs.board-descriptors }}
run: |
# MODIFY BEFORE merge vvvvv
cd ./hwci
ls
source ./.venv/bin/activate
cd ..

git clone https://github.com/treadmill-tb/tockloader.git
cd tockloader
pip install -e .
cd ..

STEP_FAIL=0

# Generate a summary of all the tests executed:
cat <<GITHUB_STEP_SUMMARY >>"$GITHUB_STEP_SUMMARY"
### <a id="tml-job-summary-${{ matrix.tml-job-id }}"></a>Tests executed on board \`nrf52840dk\`, job ID ${{ matrix.tml-job-id }}
### <a id="tml-job-summary-${{ matrix.tml-job-id }}"></a>Tests executed on job ID ${{ matrix.tml-job-id }}

| Result | Test |
|--------|------|
Expand All @@ -293,7 +308,27 @@ jobs:
echo "===== RUNNING TEST $TEST ====="
FAIL=0
set -o pipefail
python3 core/main.py --board boards/nrf52dk.py --test "$TEST" 2>&1 | tee ./job-output.txt || FAIL=1

# Check if this is a multi-board test with specific board descriptors
if [ "$MULTI_BOARD" = "true" ] && [ -n "$BOARD_DESCRIPTORS" ]; then
echo "Running multi-board test with specified board descriptors"
BOARD_DESC_ARGS="--board-descriptors"
for DESC in $BOARD_DESCRIPTORS; do
BOARD_DESC_ARGS+=" $DESC"
done

pwd

echo "python3 hwci/core/main.py $BOARD_DESC_ARGS --test hwci/$TEST 2>&1 | tee ./job-output.txt || FAIL=1"

# Run test with multiple board descriptors
python3 hwci/core/main.py $BOARD_DESC_ARGS --test "hwci/$TEST" 2>&1 | tee ./job-output.txt || FAIL=1

else
# Run standard single-board test
python3 hwci/core/main.py --board hwci/boards/nrf52dk.py --test "hwci/$TEST" 2>&1 | tee ./job-output.txt || FAIL=1
fi

set +o pipefail

# Insert the result into the markdown table:
Expand Down Expand Up @@ -328,11 +363,9 @@ jobs:
echo "At least one test failed, exiting with error."
exit 1
fi

- name: Request shutdown after successful job completion
run: |
sudo touch /run/github-actions-shutdown

- name: Provide connection information on job failure
if: failure()
run: |
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
model: nrf52840dk
hw_rev: '3.2'
serial_number: 0xfoobar
serial_number: '0679be07-6106-48aa-8057-b1d4f2e18a99'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think serial_number is a good label here. This is just the host ID, no?

board_module: hwci/boards/nrf52dk.py
features:
ble: true
debugger: jlink
gpio: true
pin_mappings:
P0.13:
io_interface: raspberrypi5gpio
Expand All @@ -26,11 +30,3 @@ pin_mappings:
target_pin_function: BUTTON2
target_pin_mode: input
target_pin_active: low
P1.01:
io_interface: raspberrypi5gpio
io_pin_spec: 16
target_pin_function: GPIO0
P1.02:
io_interface: raspberrypi5gpio
io_pin_spec: 13
target_pin_function: GPIO1
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
model: nrf52840dk
serial_number: '0af84b36-1d44-4e0e-9046-1f3fd8ec1cbf'
board_module: boards/nrf52dk.py
host_type: QEMU
features:
ble: true
debugger: jlink
gpio: false
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
model: nrf52840dk
serial_number: '1bdc10a7-9bea-4da5-9e9c-02c046223dfb'
board_module: hwci/boards/nrf52dk.py
host_type: QEMU
features:
ble: true
debugger: jlink
gpio: false
Loading
Loading