Skip to content

NOJIRA-Storage-manager-graceful-signing-degradation - #1149

Merged
pchero merged 3 commits into
mainfrom
NOJIRA-Storage-manager-graceful-signing-degradation
Jul 30, 2026
Merged

NOJIRA-Storage-manager-graceful-signing-degradation#1149
pchero merged 3 commits into
mainfrom
NOJIRA-Storage-manager-graceful-signing-degradation

Conversation

@pchero

@pchero pchero commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Make bin-storage-manager degrade gracefully instead of crash-looping when no usable GCS signing credential is configured, mirroring the transcribe-manager STT fix pattern. Found while bringing the self-hosted sandbox to full service parity (sandbox repo: docs/plans/2026-07-31-sandbox-full-service-parity-design.md, section 2.6): commit 1f90029 made a missing GOOGLE_APPLICATION_CREDENTIALS fatal at boot, killing the whole service for what is genuinely only a signing-capability gap.

  • bin-storage-manager: NewFileHandler returns a working keyless handler when GOOGLE_APPLICATION_CREDENTIALS is unset (absence degrades); a SET-but-unreadable/unparsable credential still fails startup loudly (misconfiguration stays fatal)
  • bin-storage-manager: Add unauthenticated storage-client fallback for the credential-less case only, so a fully credential-less host can still boot for read/delete/bookkeeping RPC
  • bin-storage-manager: Signing-dependent read paths (DownloadURIGet, DownloadURIRefresh, compress-file and recording fetch) return a structured SIGNING_NOT_CONFIGURED VoipbinError (Unavailable) instead of a bare error
  • bin-storage-manager: fileHandler.Create tolerates any download-URI generation failure (nil key or sign-time failure) - persists the file record with an empty URIDownload instead of failing and orphaning the already-moved GCS object; the URI populates later via DownloadURIRefresh once a real key exists
  • bin-storage-manager: Add signing-availability gauge and download-URI failure counter metrics
  • bin-storage-manager: Update CLAUDE.md, docs/architecture.md, docs/dependencies.md, docs/operations.md for the new failure modes
  • bin-api-manager: Add the SIGNING_NOT_CONFIGURED row to the RST error reference and rebuild docs

pchero added 3 commits July 31, 2026 03:15
- bin-storage-manager: Make a missing GOOGLE_APPLICATION_CREDENTIALS non-fatal; NewFileHandler returns a working handler with no signing key instead of killing the service at boot
- bin-storage-manager: Keep a configured-but-unreadable/unparsable credential path fatal, so misconfiguration stays loud
- bin-storage-manager: Fall back to an unauthenticated storage client only on a keyless deployment, since the same env var is the client's ADC source
- bin-storage-manager: Return a structured VoipbinError (UNAVAILABLE / SIGNING_NOT_CONFIGURED) from the signed-URL guard, so callers get a typed 503 instead of an opaque 500
- bin-storage-manager: Treat an empty private key or access id as not configured, not as a signer error
- bin-storage-manager: Let file Create tolerate any download-URI failure, persisting an empty URIDownload and NULL TMDownloadExpire instead of failing and orphaning the moved GCS object
- bin-storage-manager: Log the not-configured case as a warning and other signing failures as errors across Create, DownloadURIGet and DownloadURIRefresh
- bin-storage-manager: Add signing_available gauge and download_uri_failure_total counter so the degraded mode is alertable
- bin-storage-manager: Add tests for the keyless constructor, the unauthenticated client fallback, the structured guard error and both Create download-URI outcomes
- bin-storage-manager: Update CLAUDE.md, docs/architecture.md, docs/dependencies.md and docs/operations.md for the new degradation behavior
- bin-api-manager: Document the SIGNING_NOT_CONFIGURED storage reason in the RST error reference and rebuild the docs
- bin-timeline-manager: Fix migration 000004 failing on every fresh ClickHouse
  install - a semicolon inside a comment line made golang-migrate's naive
  x-multi-statement splitting produce a comment-only statement chunk, which
  ClickHouse rejects with "code: 62, Empty query", crash-looping the service at
  boot. Reworded the comment and added a warning note. Verified against a fresh
  ClickHouse 24.8: migrations now complete to version 6, dirty=false
- bin-timeline-manager: Document the one-time recovery for ClickHouse
  instances left at dirty=1 version=4 by the pre-fix migration 000004
  (append-only TinyLog schema_migrations - recovery is an INSERT of a clean
  version-3 row, verified against golang-migrate v4.19.1's driver source)
@pchero
pchero merged commit 36a0dba into main Jul 30, 2026
2 of 3 checks passed
@pchero
pchero deleted the NOJIRA-Storage-manager-graceful-signing-degradation branch July 30, 2026 19:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant