feat: implement readiness and liveness probes for bucket migration

[axelmarciano]

This pull request introduces several improvements and fixes to the migration process, Helm chart, and deployment readiness for the Expo Open OTA service. The main focus is on making bucket migrations safer and more observable, improving environment variable rendering for different deployment modes, and updating health probes to better coordinate with orchestrators. The most significant changes are grouped below.

Deployment and Migration Readiness Improvements:

• The API server now starts immediately and serves /hc (liveness) during bucket migrations, returning 503 for all other endpoints until migrations complete. This prevents the orchestrator from killing pods during long migrations and only marks them as ready when safe to serve traffic. /ready is not registered until migrations finish. (cmd/api/main.go)
• Health probes in the Helm deployment are updated: /hc is used for liveness (always 200), and /ready for readiness (only 200 after migrations), ensuring correct orchestration behavior during upgrades. (helm/templates/deployment.yaml)

Bucket Migration Parallelization and Observability:

• The S3 bucket migration now moves objects in parallel (default 16 workers, configurable via BUCKET_MIGRATION_CONCURRENCY), significantly speeding up migrations. Marker files are moved only after all other objects to ensure safe retries. Progress is logged every 500 objects to provide visibility during long migrations. (internal/bucket/v2_migration.go) [1] [2] [3]

Helm Chart Environment Variable Rendering:

• Environment variables required for both stateless and control-plane modes are now rendered according to deployment mode: required in stateless mode, optional (but present) in control-plane mode for smooth upgrades and backward compatibility. This ensures legacy variables are available for migration and v1 client support. (helm/values.yaml, helm/templates/deployment.yaml) [1] [2] [3] [4]
• Optional tuning variables (e.g., AWS_BASE_ENDPOINT, BUCKET_KEY_PREFIX) are now always rendered with enabled: true to prevent accidental omission from deployments. (helm/values.yaml) [1] [2]

Helm Chart Testing Enhancements:

• New tests verify that environment variables are rendered correctly for both stateless and control-plane modes, that optional variables are present, and that health probe paths are set as intended. (helm/helm_template_test.go) [1] [2] [3]

These changes together make migrations safer and more robust, improve deployment reliability, and ensure smooth upgrades with strong backward compatibility.

Summary by CodeRabbit

• New Features

  • Services now start responding to health checks immediately while migrations complete.
  • Readiness reporting is separated from liveness reporting: /hc remains available during startup, while /ready becomes available after migrations finish.
  • Bucket migrations now process large object sets concurrently for faster completion.

• Bug Fixes

  • Improved handling and diagnostics for missing legacy signing-key configuration.
  • Corrected rendering of optional Helm environment variables and conditional settings.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 084db49e-7185-447f-8ad2-8039fbf83ea2

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Changes

The application now starts HTTP serving before completing distributed bucket migrations, exposes migration-aware liveness and readiness behavior, concurrently moves S3/GCS objects, updates Helm environment and probe rendering, and validates missing legacy key configuration earlier.

Migration-aware startup

Layer / File(s)	Summary
Distributed migration coordination internal/bucketmigration/runner.go	Pending migrations are executed under a renewable distributed lock through EnsureMigrations.
Concurrent bucket move execution internal/bucket/v2_migration.go	S3 and GCS pass-2 moves use bounded concurrency, progress logging, error collection, and marker-last processing.
Migration-aware HTTP startup cmd/api/main.go, internal/router/router.go	The server initially serves /hc and temporary 503 responses, then atomically activates the real router after migrations; /ready is registered on that router.
Helm probes and environment rendering helm/templates/deployment.yaml, helm/values.yaml, helm/helm_template_test.go	Readiness uses /ready, liveness uses /hc, environment enablement supports booleans and conditions, and rendering tests cover optional variables and probe paths.
Legacy key migration validation internal/database/postgres/migrations/20260613110419_migrate_infra_to_db.go, internal/database/postgres/migrations/seal_legacy_keys_test.go	Empty local or environment key configuration now fails early with targeted test coverage.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant HTTPServer
  participant EnsureMigrations
  participant RealRouter
  HTTPServer->>HTTPServer: Serve /hc and temporary 503 responses
  HTTPServer->>EnsureMigrations: EnsureMigrations()
  EnsureMigrations-->>HTTPServer: Migrations complete
  HTTPServer->>RealRouter: Activate router
  RealRouter-->>HTTPServer: Serve /ready and application routes

Loading

Suggested reviewers: emacoricciati

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately reflects the main change: adding readiness and liveness behavior around bucket migrations.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/readiness-migrations-helm

---

_{Comment @coderabbitai help to get the list of available commands.}

[axelmarciano]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cmd/api/main.go`:
- Around line 48-53: Update the http.Server initialization in the main server
setup to configure nonzero ReadHeaderTimeout, ReadTimeout, and IdleTimeout
values, preventing slow or idle clients from holding connections indefinitely.
Do not add WriteTimeout, or choose a sufficiently generous value that will not
interrupt long-lived manifest or asset responses.

In `@helm/helm_template_test.go`:
- Around line 75-82: Replace every exec.Command invocation in
helm_template_test.go, including the calls in the test sections around lines 75,
140, 166, and 195, with exec.CommandContext using an appropriate test-scoped
context so the noctx linter passes. Preserve each command’s arguments and
existing execution behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 52e65025-887f-4586-846f-dcae40800c88

📥 Commits

Reviewing files that changed from the base of the PR and between afd3568 and adb0369.

📒 Files selected for processing (9)

• cmd/api/main.go
• helm/helm_template_test.go
• helm/templates/deployment.yaml
• helm/values.yaml
• internal/bucket/v2_migration.go
• internal/bucketmigration/runner.go
• internal/database/postgres/migrations/20260613110419_migrate_infra_to_db.go
• internal/database/postgres/migrations/seal_legacy_keys_test.go
• internal/router/router.go