feat(backend): security hardening + provider_api invariant + expanded API surface + alembic migrations

[fgogolli]

Description

Backend hardening for ORB: security posture, data-safety on upgrade, runtime correctness, and consolidated AWS provider contracts. 19 grouped commits, 7663 unit tests passing, zero regressions. UI + UX flow ships in a companion PR.

Highlights

Security + Auth

• Anonymous identity falls back to viewer, never admin. The previous anonymous-admin fallback is gone — auth-disabled deployments no longer hand out admin to network callers, and the server warns at startup if it binds to a non-loopback host with auth off.
• require_role("viewer") enforced on every read endpoint under /requests, /machines, /templates, /events, /me, /providers, /observability, and /system. A dynamic route-enumeration test guards against future regressions.
• check_destructive_admin_allowed is a public Pydantic-aware dependency; it refuses when auth is disabled (fail-closed) and is no longer a private symbol reached via cross-router imports.
• /providers/health no longer echoes provider error strings to the client (account IDs / ARNs stay server-side); failures log at WARNING with exc_info=True.
• /admin/init returns a generic error message with a correlation ID (fresh uuid4) — never str(exc), no filesystem or traceback leak.
• POST /config/save refuses paths that escape the configured config directory. Uses Path.resolve() + .is_relative_to() for containment; passes the resolved path to save_config so the check and the write hit the same target (TOCTOU-safe).
• Audit log middleware captures GETs to /api/v1/config, /api/v1/admin, and /api/v1/me so secret-read attempts leave a trace. X-Correlation-ID sanitized against C0 + C1 controls and U+2028/U+2029 (log-injection safe), 128-char cap.
• Logging middleware redacts token=, api_key=, access_token=, and password= query parameters before writing to debug logs.
• CSP drops unsafe-inline for scripts; HSTS only emitted when require_https=true; X-XSS-Protection removed; /info no longer reports the auth strategy name. Security headers live in an unconditional SecurityHeadersMiddleware — present regardless of auth state.
• PID file and daemon stdio log now mode 0o600. Daemon writes a per-instance bearer token to <work_dir>/server/orb-server.token (mode 0o600 at creation, no post-write chmod window).
• orb server reload sends Authorization: Bearer <token> so reload works under bearer-token auth. Token comparison uses secrets.compare_digest, ASCII-encoded with non-ASCII crash guard. Token set is a class attribute so it survives test-suite module reloads.
• Rate-limit middleware ships enabled by default (300 req/min per identity, burst 60). Shares a get_real_client_ip helper with AuthMiddleware — walks X-Forwarded-For right-to-left, skipping trusted proxies (spoof-resistant). Startup warns when workers > 1 because per-process buckets sum multiplicatively.
• Unknown role claims log a warning. X-ORB-Scheduler header only honoured for operator-or-higher roles.

Data + Upgrade Safety

• Initial Alembic migration is the sole head; ships schema + indexes on machines.status and machines.provider_api in one migration. Downgrade raises NotImplementedError (data-preserving guardrail — restore from backup).
• SQL storage strategy auto-stamps head on first boot when tables exist without an alembic_version row. Multi-worker concurrent startup serialised via BEGIN IMMEDIATE (SQLite) / SERIALIZABLE (Postgres) transaction; losers log and exit cleanly.
• RepositoryQueryError (defined in application/ports) is raised by count_by_column on SQL errors instead of silently falling through to a 100k-row list scan. Dashboard orchestrator catches it explicitly and degrades to empty counts with WARNING.
• Nullable JSON columns (tags, metadata, provider_data, security_group_ids, health_checks) coerced to empty containers before model_validate — legacy rows survive reads. Coercion runs ahead of the schema_version fast-path.
• Machine provider_api is required at the domain boundary (Field(..., min_length=1)); empty-string sentinel rows are logged at ERROR and skipped by _safe_deserialize_iter on list endpoints. Single-id lookups propagate the ValidationError so callers see a real 404.
• Deprovisioning-critical repository queries (find_by_return_request_id) use _iter_deserialized_strict — a bad row raises rather than silently dropping, so a return request cannot appear "successful" against a partially-deserialized set.
• Return-request machine_ids invariant: NULL is a ValueError at read time. Prevents the class of bug where an empty machine_ids collapses with an empty db_machines and stamps the request COMPLETED with instances still running.
• Request return status now requires positive termination evidence: COMPLETED fires only when we had machines and provider now reports none. empty AND empty returns IN_PROGRESS and continues polling.
• _safe_deserialize_iter logs at ERROR with explicit "do not wipe" guidance and a per-entity skip counter surfaced via a storage.deserialize health check.
• Drift test guards against default_factory Pydantic fields vs nullable SQL columns.
• orb storage migrate stamp <revision> exposes alembic stamp via the CLI.

Runtime Correctness

• Fleet release policy — compute_fleet_release_decision now takes a FleetCapacityInput dataclass carrying weighted capacity (instance_weighted_capacity_units), not just headcount. Release managers read WeightedCapacity from AWS describe_fleet_instances / describe_spot_fleet_instances and pass it through, so weighted templates (e.g. t3.medium weight=4) no longer trigger false-full-return arithmetic.
• Request-type fleets are never cancelled by partial-return logic alone. Primary cancel path is gated on is_full_return AND requires_capacity_reduction; for request fleets a positive _fleet_has_no_remaining_instances describe is required. Fixes the cancelled_running state observed under weighted partial return.
• Handler contracts — fulfillment_final is renamed to intent-specific keys: requires_async_polling (handler → provisioning orchestrator) and fulfillment_complete (strategy → status polling). Each has a documented truth table; no more inverse-meaning collisions.
• Typed FleetCapacityFulfilment dataclass returned by _fetch_ec2_fleet_capacity / _fetch_spot_fleet_capacity / _fetch_asg_capacity. Downstream consumers use attribute access, not .get() with string keys.
• BaseFleetReleaseManager consolidates the shared release flow across EC2Fleet + SpotFleet. Legitimate divergence preserved: delete_fleets vs cancel_spot_fleet_requests, tag keys.
• _resolve_provider_api consolidated in the AWS handler base with a documented priority order and per-handler _default_provider_api() hook.
• ASG marked requires_async_polling=True (create returns group ID, not instances) so the polling loop observes instance state transitions before stamping COMPLETED.
• Machine.from_provider_format populates both provider_api and provider_name from snake_case or camelCase keys; raises ValueError with a clear message when either is missing (no more silent Pydantic ValidationError landmine).
• DashboardSummaryOrchestrator opens a single UoW covering counts + recent activity so all figures reflect one snapshot.
• Dashboard sources by-status counts from count_by_status / count_by_provider_api GROUP BY queries instead of listing with limit=100_000 against handlers that clamp at 1000.
• Requests router drops the hand-maintained terminal-status set in favour of RequestStatus.is_terminal() so timeout state closes the SSE stream.
• Template list handler runs is_active against domain DTOs before filter_expressions converts items to dicts.
• cleanup_database treats rows with NULL created_at as too-recent-to-purge instead of letting them slip past the cutoff.
• wipe_database slow path skips entities with no primary key the same way the fast path does.
• system_handlers.__init__ no longer double-assigns super().__init__ and self.container.

IPC + Async Hygiene

• reload-config classified as non-destructive — admin role alone is sufficient; no allow_destructive_admin=True required.
• _loopback_reload_request runs via asyncio.to_thread so the async CLI handler no longer blocks the event loop.
• handle_storage_migrate awaits create_subprocess_exec and scrubs the DB URL password from captured output.
• Daemon double-fork keeps the PID lock fd held continuously across parent → intermediate → grandchild so two simultaneous starts can't slip into the gap.
• WipeDatabaseService.execute, CleanupDatabaseService.bulk_cleanup, and open(config_file) all wrapped in run_in_executor — long-running admin routes no longer block the API event loop.
• ListActiveRequests concurrent sync wraps each per-request AWS call in asyncio.wait_for(timeout=30s) (configurable) so one stuck DescribeInstances can't hang a whole page. gather uses return_exceptions=True with an explicit re-raise pass for BaseException subclasses (CancelledError, KeyboardInterrupt).
• repository.save() no longer creates a throwaway event loop with asyncio.run when called outside a running loop; it logs a warning and skips the publish so SSE subscriber queues stay valid.
• Server startup warns when uvicorn workers > 1 and the events router is registered (SSE pubsub is single-process).

SSE

• _SseEventBus._subscribers is a set protected by an asyncio.Lock; concurrent subscribe/unsubscribe cannot corrupt the set. unsubscribe is O(1) via discard.
• Every event carries a monotonic sequence_id. history_since_seq(since_seq) emits a replay_truncated sentinel when the deque cannot serve the requested history — including after server restart (empty deque + non-zero since_seq) and when the client claims a seq_id ahead of anything issued this process lifetime.
• SSE history backed by collections.deque(maxlen=512) so publishes no longer pay an O(n) slice cost.

CI + Ops + Docs

• Deployment docs warn that systemd / launchd unit files must use --foreground; daemon double-fork would otherwise look like immediate exit to the supervisor.
• rxconfig.py lives at src/orb/ui/rxconfig.py and ships in the wheel; embedded-UI runtime points reflex run at that directory. Repo-root re-export keeps local dev workflows working.
• CHANGELOG captures the new orb-server.token file, the allow_destructive_admin field, and the initial Alembic migration.
• server_daemon.tail_log docstring no longer claims automatic rotation; deployment docs include a logrotate copytruncate sample.
• ci-tests gained a providers-tests-serial job so tests marked @pytest.mark.serial actually run on every PR.
• CI workflows now carry a top-level concurrency: block so a fresh push cancels in-flight runs; the auto-format push job opts out.
• pytest CI variants use -n 2 --dist=loadscope (matches GitHub runner CPU count).
• e2e job pairs continue-on-error with an explicit failure annotation in GITHUB_STEP_SUMMARY.
• orb.audit logger gains a dedicated rotating file handler when an audit log path is configured.
• .gitignore extended to exclude per-package coverage-*.xml artifacts.

Test Coverage

• New router tests for events, me, observability, providers, system.
• test_role_enforcement.py dynamically enumerates every route with a require_role dependency and asserts anonymous → 403. A floor guard (>= 7) prevents the enumeration from silently returning empty on future refactors.
• DashboardSummaryOrchestrator: happy / empty / sub-orchestrator-raises paths, recent_activity cap, ISO helpers on None, count-by-status fast paths, single-UoW snapshot invariant.
• server_command_handlers: start/stop/status/restart/reload/logs all unit-tested with the daemon module mocked; reload covers loopback success, loopback failure → SIGHUP fallback, ValueError fallback, bearer-token header presence vs absence.
• server_runtime: signal handler installation, SIGHUP reload semantics (asserts cm.reload is actually called), embedded entrypoint wiring of cwd + env + signal forwarding.
• AuditLogMiddleware and ReadOnlyMiddleware full dispatch-matrix tests; audit log verified to strip credentials from query strings.
• _safe_deserialize_iter: skip-on-error semantics, ERROR-level log, per-entity skip counter exposure.
• Live capacity helper: assert_terminal_ok accepts complete_with_error partials when fulfilled_units >= 1 AND message names the shortfall (regex anchored to canonical AWS prefixes); rejects 0/0 bogus responses; retries describe_instances on InvalidInstanceID.NotFound for AWS control-plane propagation.
• Multi-* termination tests assert fulfilled_units >= target_units (weighted-capacity contract), not headcount.
• Fleet release policy unit tests cover weighted arithmetic + primary-cancel gating + _fleet_has_no_remaining_instances describe verification.
• Consolidated _resolve_provider_api covered across all 4 handlers.

Other Polish

• _SseEventBus history backed by collections.deque(maxlen) so publishes no longer pay an O(n) slice cost.
• Rate-limiting promoted from Optional[dict] to a real RateLimitConfig Pydantic model; middleware consumes the typed shape; default config files seeded with a safe rate-limit block.
• Audit log uses one wall clock for ts and a separate monotonic delta for latency.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change
• Documentation update
• Performance improvement
• Code cleanup or refactor
• Dependencies update
• CI/CD or build process changes

Related Issues

N/A — post-audit follow-up; no linked issue.

How Has This Been Tested?

• Unit tests added/updated
• Integration tests added/updated
• Manual testing performed
• AWS live tests run against the branch

Local: pytest tests/unit tests/api tests/infrastructure tests/integration tests/interface tests/providers/aws/unit → 7663 passed, 88 skipped, 0 failed. Pyright clean on all touched files. Ruff lint + format clean. Bootstrap smoke test (register core + server services on a fresh DI container) boots clean. Alembic auto-stamp tested against a pre-existing SQLite database without alembic_version. Full AWS live-test suite run against the branch; remaining transient failures fixed by describe_instances retry + positive-termination-evidence guard.

Test Configuration

• Python version: 3.12
• OS: macOS (darwin) for development; CI runs Linux 3.10 / 3.11 / 3.12 / 3.13 matrix
• Dependencies changed: Yes — adds alembic, sqlalchemy, psutil. The ui extra is not added in this PR; it ships with the companion UI PR.

Checklist

• Code follows the project's style guidelines
• Self-review performed
• Comments in hard-to-understand areas
• Documentation updated where applicable
• No new warnings introduced
• Tests added that prove the fix / feature
• New and existing unit tests pass locally
• Dependent changes merged and published
• CHANGELOG updated

Performance Impact

• Performance improved

• ListActiveRequests parallel sync (Semaphore(8)) with per-task 30s timeout — a 50-request page no longer pays 50× the AWS round-trip latency and one stuck call can't hang the whole page.

• MachineSerializer schema_version fast-path skips legacy fixup on every hot read (now with safe NULL coercion ahead of the fast-path).

• Dashboard by-status counts use SQL GROUP BY instead of listing 100k rows and bucketing in Python.

• SSE history switched from O(n)-per-publish list slicing to deque(maxlen=...) constant-time append.

• Indexes on machines.status and machines.provider_api land with the initial schema.

• Admin wipe / cleanup and blocking config-file reads offloaded to run_in_executor — the API event loop stays responsive during long-running admin operations.

Security Considerations

• Security improved

A default install no longer hands out admin to anonymous network callers, secret-bearing config endpoints require admin, audit logs capture sensitive reads without leaking credentials from URLs or X-Correlation-ID, path-traversal on /config/save is refused (TOCTOU-safe), the destructive-admin gate fails closed under auth-disabled, loopback token comparison is constant-time and ASCII-safe, rate limiting is on by default with proxy-aware IP resolution, and security headers are emitted unconditionally regardless of auth state.

Dependencies

• alembic, sqlalchemy (new) — required by the SQL storage backend + migration.
• psutil (new) — required by the system health check (CPU/memory).
• The ui / reflex extra is intentionally not added here; it ships with the companion UI PR.

Migration Guide

Existing JSON backend deployments

No action required.

Existing SQL backend deployments (pre-Alembic)

The SQL strategy auto-stamps head on first boot when tables exist without an alembic_version row. No manual alembic stamp needed for the common case.

For explicit control:

orb storage migrate stamp <head-revision>
orb storage migrate up

Machines with provider_api NULL or empty appear as skipped rows in the health probe (storage.deserialize); fix by updating the row or re-running the source-request backfill query documented in the migration comments.

Reload IPC

POST /admin/reload-config is the single reload endpoint; callers of the old POST /config/reload need to update.

Loopback admin token

The daemon writes <work_dir>/server/orb-server.token on start (mode 0o600). The reload CLI reads it for Authorization: Bearer … against the API. No manual setup required; the file is auto-cleaned on stop.

Rate limiting

Now enabled by default. If your deployment relies on unrestricted request volume, set rate_limiting.enabled=false in your config, or raise requests_per_minute and burst.

Deployment Notes

• New configuration keys land in config/default_config.json (rate_limiting, allow_destructive_admin, audit_log_file, etc.). Deployments with a pinned config file should merge in the new keys before rollout.
• Under systemd / launchd, use orb server start --foreground with Type=simple. The daemon form (no --foreground) double-forks and detaches and will look like an immediate exit to the service manager.
• nginx in front of SSE requires proxy_buffering off on the /api/v1/events/ location — sample block is in docs/root/deployment/traditional.md.
• With workers > 1 the rate limiter's per-process buckets sum multiplicatively; scale requests_per_minute accordingly or move to a shared backend before deploying multi-worker.

Reviewers

@finos/open-resource-broker-maintainers

[github-actions]

Test Results Summary

    6 files  ±  0      6 suites  ±0   12m 10s ⏱️ - 6m 15s
8 372 tests +756  8 075 ✅ +543  274 💤 +190  23 ❌ +23 
8 372 runs  +384  8 075 ✅ +171  274 💤 +190  23 ❌ +23 

For more details on these failures, see this check.

Results for commit 7a8f1c6. ± Comparison against base commit 3738fee.

This pull request removes 25 and adds 781 tests. Note that renamed tests count towards both.

tests.integration.test_security_features.TestAuthMiddlewareSecurityHeaders ‑ test_all_eight_security_headers_present
tests.integration.test_security_features.TestAuthMiddlewareSecurityHeaders ‑ test_content_security_policy_present
tests.integration.test_security_features.TestAuthMiddlewareSecurityHeaders ‑ test_excluded_path_exact_match_only
tests.integration.test_security_features.TestAuthMiddlewareSecurityHeaders ‑ test_path_normalization_prevents_traversal
tests.integration.test_security_features.TestAuthMiddlewareSecurityHeaders ‑ test_permissions_policy
tests.integration.test_security_features.TestAuthMiddlewareSecurityHeaders ‑ test_referrer_policy
tests.integration.test_security_features.TestAuthMiddlewareSecurityHeaders ‑ test_strict_transport_security
tests.integration.test_security_features.TestAuthMiddlewareSecurityHeaders ‑ test_x_content_type_options_nosniff
tests.integration.test_security_features.TestAuthMiddlewareSecurityHeaders ‑ test_x_frame_options_deny
tests.integration.test_security_features.TestAuthMiddlewareSecurityHeaders ‑ test_x_xss_protection
…

tests.infrastructure.storage.migrations.test_auto_stamp_race.TestAutoStampConcurrentWorkers ‑ test_exactly_one_row_after_concurrent_stamp
tests.infrastructure.storage.migrations.test_auto_stamp_race.TestAutoStampConcurrentWorkers ‑ test_revision_is_correct_after_concurrent_stamp
tests.infrastructure.storage.migrations.test_auto_stamp_race.TestAutoStampConcurrentWorkers ‑ test_workers_do_not_raise_on_contention
tests.infrastructure.storage.migrations.test_auto_stamp_race.TestAutoStampSingleWorker ‑ test_already_stamped_db_is_skipped
tests.infrastructure.storage.migrations.test_auto_stamp_race.TestAutoStampSingleWorker ‑ test_second_call_is_no_op
tests.infrastructure.storage.migrations.test_auto_stamp_race.TestAutoStampSingleWorker ‑ test_stamps_head_revision
tests.infrastructure.storage.migrations.test_auto_stamp_race.TestAutoStampSingleWorker ‑ test_stamps_known_head_revision
tests.infrastructure.storage.test_repository_query_error.TestCountByColumnRaises ‑ test_cause_is_original_sqla_error
tests.infrastructure.storage.test_repository_query_error.TestCountByColumnRaises ‑ test_non_sqla_exception_propagates_unchanged
tests.infrastructure.storage.test_repository_query_error.TestCountByColumnRaises ‑ test_raises_repository_query_error_on_sqla_error
…

♻️ This comment has been updated with latest results.

[canonicalname]

Approved as discussed