Gateway improvements v2#281
Draft
mniestroj wants to merge 10 commits into
Draft
Conversation
Read CONFIG_EXAMPLE_FW_UPDATE_COMPONENT from the DUT's generated .config instead of hardcoding "ci_ota_fw" in the ota_firmware fixture, so the pytest side always agrees with whatever component name the C side was actually built with. The lookup uses the runners.core.BuildConfiguration helper from zephyr/scripts/west_commands and twister_harness.helpers.domains_helper to pick the default sysbuild domain's build directory. Signed-off-by: Marcin Niestroj <marcin.niestroj@canonical.com>
The refactor to the pouch CoAP stack replaced the previously unbounded k_fifo_put() used to hand blocks from the CoAP producer to the pouch consumer with a fixed-size pouch_msgq_put() called with POUCH_NO_WAIT. Whenever the BLE bearer could not drain fast enough to keep up with the CoAP block delivery rate, the very next block_cb returned -ENOMEM, which in turn made pouch_gateway_cloud_forward_pouch() propagate the error and abort the entire downlink transfer. This is easy to hit with real workloads such as an OTA firmware download proxied to a BLE peripheral, where the incoming CoAP rate is fast compared to the outgoing GATT throughput. Restore the pre-rewrite semantics by waiting up to CONFIG_POUCH_GATEWAY_DOWNLINK_BLOCK_TIMEOUT seconds for a free slot in the queue, the same timeout that is already used to acquire a buffer from the pouch_buf pool a few lines above. The CoAP producer thread now applies backpressure to the transport while the consumer (BLE bearer) catches up, instead of failing the whole transfer on the first burst. Fixes: 3fc341c ("gateway: rewrite over pouch coap stack") Signed-off-by: Marcin Niestroj <marcin.niestroj@canonical.com>
Mirror what coap_client and http_client already do: add a second
firmware-update mode that hashes the incoming OTA payload with SHA256 and
logs the digest instead of writing the data to the MCUboot slot and
rebooting.
Introduce an EXAMPLE_FW_UPDATE_MODE choice between:
* EXAMPLE_FW_UPDATE_FLASH
* EXAMPLE_FW_UPDATE_SHA256_VERIFY
Extend the CMakeLists.txt to pick the matching source file. The verifier
itself is a straight copy of coap_client/src/fw_update_test.c with the same
log lines ("OTA computed SHA256: <hex>") that the shared pytest OTA harness
already relies on.
Signed-off-by: Marcin Niestroj <marcin.niestroj@canonical.com>
Add a new pouch.gateway.ota twister scenario that exercises the
Golioth OTA download path end-to-end through the gateway's Pouch BLE
tunnel, using the ble_gatt peripheral (already spawned as a sysbuild
sub-image) as the OTA target device.
The scenario builds the peripheral with the SHA256-verify OTA mode
introduced in the previous commit, so it hashes the downloaded payload
and logs "OTA computed SHA256: <hex>" instead of touching MCUboot slots,
allowing the test to run on nrf52_bsim without a real bootloader flow.
The new test_ota.py mirrors the shared examples/zephyr/pytest/test_ota.py
that coap_client and http_client use: wait for "Credentials loaded",
wait for the SHA256 line, and compare against the digest returned by the
ota_firmware fixture. Because the gateway/pytest/ tree is separate from
the shared examples/zephyr/pytest/ tree (it overrides the dut and
creds_dir fixtures to accommodate the sysbuild BSIM layout), the
test_id, artifacts_to_cleanup, cohort, and ota_firmware fixtures are
copied verbatim, with two small adjustments:
- fw_update_component reads CONFIG_EXAMPLE_FW_UPDATE_COMPONENT from
the peripheral image's build (via runners.core.BuildConfiguration
on <build_dir>/peripheral_ble_gatt_example_0) rather than from the
default sysbuild domain the shared fixture uses, because the OTA
target is not the top-level gateway image.
- ota_firmware generates an 8 KiB payload instead of 400 KiB. The
BLE-tunneled OTA path is much slower than the direct coap/http_client
download, and 8 KiB is enough to force multi-block delivery through
the pouch tunnel while staying well within the twister timeout.
The peripheral is configured with a CI-only overlay
(overlay-ble_gatt-ci-ota.conf) applied via
peripheral_ble_gatt_example_0_EXTRA_CONF_FILE=... in sample.yaml. This
sets CONFIG_EXAMPLE_FW_UPDATE_COMPONENT="ci_ota_fw", matching the
package name coap_client and http_client already use for their CI OTA
artifacts. Overriding via an EXTRA_CONF_FILE rather than a plain
sample.yaml Kconfig assignment is required because twister strips the
outer quotes from sysbuild-image-scoped extra_args entries that don't
start with CONFIG_ / SB_CONFIG_ (see
zephyr/scripts/pylib/twister/twisterlib/runner.py:1711), which would
turn the string into a malformed Kconfig literal.
The two existing settings tests grow a pytest_root pointing at
test_sample.py so twister does not additionally try to collect and run
test_ota.py under them.
Timeouts are declared per test rather than shared through common:
lesc and auth keep their existing 180 s budget (they finish in ~35 s);
the OTA scenario gets 300 s to accommodate the multi-block BLE-tunneled
download.
Signed-off-by: Marcin Niestroj <marcin.niestroj@canonical.com>
…test
Move all fixtures and both tests into examples/zephyr/pytest, and drive
the gateway-vs-direct differences from focused pytest options passed
from each sample.yaml instead of duplicating fixtures per example.
The consolidated conftest exposes three options:
--gateway-cloud create the gateway cloud identity fixture
--target-domain sysbuild sub-image that is the Pouch DUT
--ota-image-size OTA payload size in bytes (default 400 KiB)
sample.yaml usage:
- gateway / gateway_custom_connect pass --gateway-cloud and
--target-domain=peripheral_ble_gatt_example_0.
- pouch.gateway.ota additionally passes --ota-image-size=8192
because BLE-tunneled OTA is much slower than direct downloads.
- coap_client and http_client pass no pytest_args and rely on
defaults.
Notable fixture changes:
- target_build_dir picks the sysbuild sub-image build tree; both
creds_dir and build_conf derive from it.
- dut always uses 'west flash -d <build>', which works for the plain
native_sim binary just like it does for the gateway's BabbleSim
sysbuild layout (which spawns every sub-image at once).
- gateway_creds signs the gateway cert against the shared CA already
present in creds_dir; skipped when --gateway-cloud is off.
- setup deletes LED settings on [device, gateway] with None filtered
out, so direct-client tests reuse the same cleanup path.
- settings_device becomes 'gateway or device'.
- BuildConfiguration for the OTA target sub-image goes through
target_build_dir so it stays correct on both native_sim and
BabbleSim sysbuild layouts.
- pytest_dut_scope: module is set inside each scenario's
harness_config because twister replaces (not deep-merges) that
map.
Drop the previous --gateway-name / GOLIOTH_GATEWAY_NAME override: no CI
workflow or doc exercises it, and the gateway fixture now always
generates a fresh cloud identity per test run.
Verified with:
west twister -c -v -p nrf52_bsim/native -T pouch/examples/zephyr/gateway -W
west twister -c -v -p native_sim/native -T pouch/examples/zephyr/http_client -W
west twister -c -v -p native_sim/native -T pouch/examples/zephyr/coap_client -W
west twister --dry-run -p nrf52_bsim/native -T pouch/examples/zephyr/gateway_custom_connect
…als helper
Split the CA generation and the device-cert-signing logic out of the
`creds` fixture so downstream conftests can reuse them.
- New `ca` module-scope fixture generates ca.key.pem, ca.crt.pem and
ca.der in creds_dir and returns their paths.
- New `generate_device_credentials(dest_dir, device_name, project_id,
ca_key, ca_cert, *, crt_der_name="crt.der", key_der_name="key.der")`
helper signs a device key + cert against the shared CA and writes
the device DER files into dest_dir under configurable names.
- `creds` now depends on `ca` and delegates the device-cert generation
to generate_device_credentials; it still uploads the CA cert to
Golioth as before.
The only observable change for existing consumers is an extra ca.der
file in creds_dir, which is unused today but ready for firmware images
that want to load the CA from the filesystem.
Signed-off-by: Marcin Niestroj <marcin.niestroj@canonical.com>
…tials Rewire gateway_creds to reuse the shared `ca` fixture and generate_device_credentials helper introduced in the previous commit instead of inlining the openssl invocations. The dependency direction becomes: ca (CA private key + cert; from plugin) |-- creds (peripheral cert; from plugin) \`-- gateway_creds (gateway cert; local) gateway_creds no longer depends on `creds`, so the two device-cert fixtures are peers instead of one shadowing the other. The gateway build directory is still separate from `creds_dir` today, so gateway_creds copies the CA DER file from `creds_dir` into `gateway_dir` for the firmware to load. Signed-off-by: Marcin Niestroj <marcin.niestroj@canonical.com>
Add EXAMPLE_POUCH_DEVICE_CRT_FILENAME and EXAMPLE_POUCH_DEVICE_KEY_FILENAME Kconfig strings (defaulting to "crt.der" and "key.der") and consume them in credentials.c instead of hardcoding the filenames. No behavior change with the defaults; the knobs let a caller point a single shared credentials directory at per-device filenames later. Signed-off-by: Marcin Niestroj <marcin.niestroj@canonical.com>
…DER filenames Add EXAMPLE_POUCH_DEVICE_CRT_FILENAME / EXAMPLE_POUCH_DEVICE_KEY_FILENAME Kconfig strings (defaulting to "crt.der" / "key.der") in both gateway/Kconfig and gateway_custom_connect/Kconfig, and consume them in gateway/src/credentials.c (shared with gateway_custom_connect via CMakeLists) instead of hardcoding the filenames on the load_certificate / load_private_key (pouch identity) path. The parallel Kconfig knobs for the gateway mTLS identity (EXAMPLE_COAP_CLIENT_GW_DEVICE_CRT/KEY_FILENAME) already existed and are untouched here. No behavior change with the defaults; the knobs let a shared credentials directory address pouch identities by per-device filenames later. Signed-off-by: Marcin Niestroj <marcin.niestroj@canonical.com>
…-device DER filenames
Route both peripheral and gateway sub-images at a single shared
credentials directory so all device certs live next to a single
CA, keyed by per-device DER filenames.
Sysbuild side:
- SB_CONFIG_PERIPHERAL_MOUNT_CREDS and SB_CONFIG_GATEWAY_MOUNT_CREDS
change from bool to string: they now carry the host-side directory
mounted at /creds inside each sub-image. Empty disables the mount.
- gateway/sysbuild.cmake and gateway_custom_connect/sysbuild.cmake
consume the string in the -volume=... cmdline arg instead of the
hardcoded "creds" path, and derive the pouch identity + gateway
mTLS DER filenames from the sub-image name (e.g.
"peripheral_ble_gatt_example_0_crt.der", "gateway_crt.der"). This
keeps the sample.yaml files small: each scenario only sets the
mount source once.
- The gateway and gateway_custom_connect sample.yaml files set both
strings to "../creds", so each sub-image (whose sim runtime cwd is
its own build tree) resolves the mount at <build>/creds/.
Pytest side:
- Drop the local creds_dir override; creds_dir now resolves to
<build>/creds via the plugin default, shared by both sub-images.
- Override the plugin creds fixture locally so it reads the pouch
identity DER filenames from the target sub-image build_conf and
passes them to generate_device_credentials; falls back to the
plugin defaults (crt.der / key.der) when the Kconfigs are absent
(direct-client samples).
- Add gateway_build_dir / gateway_build_conf fixtures and update
gateway_creds to write into the shared creds_dir under gateway
filenames read from the gateway sub-image build_conf. ca.der is
already there from the ca fixture, so gateway_creds no longer
copies it.
Signed-off-by: Marcin Niestroj <marcin.niestroj@canonical.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.