Skip to content

Build-context upload to cloud storage intermittently fails (region-scoped; eu-central blocked ~45 min on 22 Jul) + SDK logs empty error detail #186

Description

@mannyb223

Summary

pipecat cloud deploy (and the pipecatcloud 0.7.2 module path) intermittently fails at the build-context upload step with Failed to upload build context to cloud storage. It has been an occasional single-retry flake for us since ~3 Jul 2026, but on 22 Jul it hard-blocked our eu-central deploys for ~45 minutes (6 consecutive failures) while us-west deploys of the same context succeeded — and because cross-region build promotion is rejected (Build is in region 'us-west' but service is in region 'eu-central'), there is no workaround when this happens: eu services simply cannot be updated until the upload path recovers.

Timeline (22 Jul 2026, times UTC+7)

  • 17:56 us-west deploy: upload + build OK
  • 19:19, 19:25, 19:32, 19:38, 19:43 eu-central deploys: upload_to_s3:382 - Upload error: (empty detail) → Failed to upload build context to cloud storage, 5/5
  • 19:15 (same window) us-west deploy of the SAME repo state: OK — failure was region-scoped
  • Attempted workaround: promote the successful us-west build to eu-central via --build-id → rejected with Build is in region 'us-west' but service is in region 'eu-central'. Builds must be in the same region as the service.
  • 20:52 eu-central: OK first try (recovered)
  • 21:01 eu-central (different agent, same secret-set region): failed again → recovered on a later retry

Context size is well under the 500MB cap (repo build context ~a few MB, excludes applied). daily.co status page showed all-operational throughout.

Two asks

  1. Whatever intermittently 4xx/5xxs or times out on the presigned build-context upload for eu-central — visibility/fix. If you can see server-side logs for our org (ambitions-ai) in the window above, that should pinpoint it.
  2. SDK papercut that makes this much harder to diagnose: pipecatcloud/_utils/build_utils.py upload_to_s3 logs logger.error(f"Upload error: {e}") — in every one of our failures the rendered message is empty (Upload error: with nothing after it), i.e. str(e) of the exception is blank (typical for some aiohttp exceptions). Logging repr(e) / type(e).__name__ (and the HTTP status/body branch already does this better) would let users report the actual cause instead of "empty error".

Environment

  • pipecatcloud 0.7.2 (module path), macOS, org ambitions-ai
  • Same behaviour through deploy.sh wrapper and direct python -m pipecatcloud deploy

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions