Symptom
CI builds intermittently fail in the pioarduino/espidf envs with:
PackageException: Got the unrecognized status code '403' when downloaded
https://components.espressif.com/api/downloads/?object_type=component&object_id=dbc87006-...
This blocked the v3.4.0 release three times (#967 once, #969 twice) before clearing on retry. With the default matrix fail-fast, a single 403 cancels the other ~9 jobs, so the whole matrix has to be re-run.
Why the direct-download URL exists
platformio.ini pulls esp_websocket_client via a direct tarball URL in the [pioarduino] and [espidf] sections:
esp_websocket_client=https://components.espressif.com/api/downloads/?object_type=component&object_id=dbc87006-...
Background:
esp_websocket_client is an ESP-IDF component (part of esp-protocols), used directly in src/sensesp/signalk/signalk_ws_client.cpp (esp_websocket_client_init/start/send_text/...).- It is not published on the PlatformIO library registry (
pio pkg search esp_websocket_client returns only unrelated packages), so it can't be referenced by registry name. The direct ESP-Component-Registry tarball URL was the available mechanism.
- The official
[arduino] env (espressif32 @ ^6.9.0) lib_ignores it (the framework bundles it); only the SSL-enabled [pioarduino]/[espidf] envs (SENSESP_SSL_SUPPORT=1) pull it explicitly.
Why it hits CI specifically
- The
components.espressif.com/api/downloads/?object_id=... endpoint returns HTTP 200 from residential IPs but 403 from GitHub Actions runner IP ranges (rate-limit / cloud-IP blocking). End users building locally are largely unaffected.
- CI caches
~/.cache/pip and ~/.platformio/.cache, but URL-based lib_deps install into the per-project .pio/libdeps, which is not cached — so the tarball is re-downloaded from the rate-limited endpoint on every run.
Proposed fixes (tiered)
Tier 1 — CI resilience (cheap, immediate):
- Set
strategy.fail-fast: false on the build/test matrices so one transient 403 stops nuking the entire matrix (a rerun then only re-runs the 1-2 genuinely affected jobs).
- Wrap the
pio ci invocation in ci/run-ci.sh with a 2-3 attempt retry so a transient 403 self-heals within the job.
Tier 2 — remove the fragile dependency (robust):
- Mirror the component tarball to a stable, org-controlled URL (e.g. a release asset under SignalK/SensESP or hatlabs) and point
lib_deps there. Covers both pioarduino and espidf envs, eliminates the rate-limited endpoint entirely.
- Or migrate the
espidf env to the IDF Component Manager via idf_component.yml (espressif/esp_websocket_client), which uses the component manager's endpoints and is the canonical mechanism. Note this does not cover the pioarduino (arduino-framework) env, so the mirror is the broader fix.
Recommend Tier 1 immediately (it would have made all three release-blocking incidents non-events), plus Tier 2 (mirror) to remove the root cause.
Note
A stray untracked src/idf_component.yml (dependencies: {idf: '>=5.1'}) exists in the tree — possibly a partial start toward the component-manager path. Worth resolving as part of Tier 2.
Symptom
CI builds intermittently fail in the
pioarduino/espidfenvs with:This blocked the v3.4.0 release three times (#967 once, #969 twice) before clearing on retry. With the default matrix
fail-fast, a single 403 cancels the other ~9 jobs, so the whole matrix has to be re-run.Why the direct-download URL exists
platformio.inipullsesp_websocket_clientvia a direct tarball URL in the[pioarduino]and[espidf]sections:Background:
esp_websocket_clientis an ESP-IDF component (part ofesp-protocols), used directly insrc/sensesp/signalk/signalk_ws_client.cpp(esp_websocket_client_init/start/send_text/...).pio pkg search esp_websocket_clientreturns only unrelated packages), so it can't be referenced by registry name. The direct ESP-Component-Registry tarball URL was the available mechanism.[arduino]env (espressif32 @ ^6.9.0)lib_ignores it (the framework bundles it); only the SSL-enabled[pioarduino]/[espidf]envs (SENSESP_SSL_SUPPORT=1) pull it explicitly.Why it hits CI specifically
components.espressif.com/api/downloads/?object_id=...endpoint returns HTTP 200 from residential IPs but 403 from GitHub Actions runner IP ranges (rate-limit / cloud-IP blocking). End users building locally are largely unaffected.~/.cache/pipand~/.platformio/.cache, but URL-basedlib_depsinstall into the per-project.pio/libdeps, which is not cached — so the tarball is re-downloaded from the rate-limited endpoint on every run.Proposed fixes (tiered)
Tier 1 — CI resilience (cheap, immediate):
strategy.fail-fast: falseon the build/test matrices so one transient 403 stops nuking the entire matrix (a rerun then only re-runs the 1-2 genuinely affected jobs).pio ciinvocation inci/run-ci.shwith a 2-3 attempt retry so a transient 403 self-heals within the job.Tier 2 — remove the fragile dependency (robust):
lib_depsthere. Covers bothpioarduinoandespidfenvs, eliminates the rate-limited endpoint entirely.espidfenv to the IDF Component Manager viaidf_component.yml(espressif/esp_websocket_client), which uses the component manager's endpoints and is the canonical mechanism. Note this does not cover thepioarduino(arduino-framework) env, so the mirror is the broader fix.Recommend Tier 1 immediately (it would have made all three release-blocking incidents non-events), plus Tier 2 (mirror) to remove the root cause.
Note
A stray untracked
src/idf_component.yml(dependencies: {idf: '>=5.1'}) exists in the tree — possibly a partial start toward the component-manager path. Worth resolving as part of Tier 2.