feat: production readiness hardening

Security: add auth to all container routes, production env validation
(fails startup if secrets missing), tighten CORS methods/headers,
catch-all exception handler, fix OAuth redirect to use APP_BASE_URL.

Infrastructure: complete production docker-compose with docker socket,
skills mount, health checks, resource limits, required POSTGRES_PASSWORD.
Non-root API container, multi-worker uvicorn, nginx security headers.

Resilience: graceful shutdown stops running containers, boot-time
reconciliation marks stale sessions as FAILED, configurable container
TTL via CONTAINER_TTL_SECONDS setting.

Observability: health check verifies DB connectivity (503 if degraded),
complete .env.example with all production variables documented.

Author: mariuspruvot

.env.example

@@ -6,6 +6,8 @@ DATABASE_URL=postgresql+asyncpg://helprs:helprs@localhost:5432/helprs
 SECRET_KEY=
 # Generate with: python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
 FERNET_KEY=
+# Required in production (ENVIRONMENT=production)
+ADMIN_PASSWORD=
 
 # GitHub App
 GITHUB_APP_ID=
@@ -22,3 +24,20 @@ VITE_API_URL=http://localhost:8000
 
 # Public URL used by the backend to build user-facing links (e.g. PR-comment session links)
 APP_BASE_URL=http://localhost:5173
+
+# CORS — JSON array of allowed origins
+CORS_ORIGINS=["http://localhost:5173"]
+
+# Environment: "development" or "production"
+# Production enforces non-empty secrets and disables admin auto-auth
+ENVIRONMENT=development
+
+# Container orchestration
+CONTAINER_TTL_SECONDS=900
+UVICORN_WORKERS=4
+
+# Docker host path to skills/ directory (required for Docker-in-Docker volume mounts)
+# SKILLS_HOST_PATH=/absolute/path/to/skills
+
+# Postgres credentials (used by docker-compose.prod.yml)
+# POSTGRES_PASSWORD=

CLAUDE.md

@@ -56,6 +56,9 @@ infra/
 - **Admin panel**: SQLAdmin at `/admin`, configured in `admin/views.py`
 - **Dashboard**: user-facing installation management at `/installations` -- installation list, session history, session replay. Authenticated users redirect from `/` to `/installations`. SQLAdmin remains at `/admin` as superadmin escape hatch.
 - **Cross-module queries**: installation module queries `ContainerSession` model directly (inline import in service functions) for session counts and lists. This avoids circular imports while keeping the API surface on the installation router.
+- **Auth on all REST routes**: identity and installation routers use `Depends(get_current_user)`, container router uses it too. The webhook handler bypasses REST routes entirely — it calls `create_session()` directly (DB record only, no container start). Container start happens when the authenticated frontend calls the REST endpoint.
+- **Production env validation**: `Settings` has a `model_validator` that enforces non-empty secrets when `ENVIRONMENT=production`. Tests use `ENVIRONMENT=test` to skip this.
+- **Graceful lifecycle**: lifespan reconciles stale RUNNING/PENDING sessions on boot (marks FAILED), and stops all running containers on shutdown. Periodic cleanup uses configurable `CONTAINER_TTL_SECONDS` from settings.
 
 ## Skills
 
@@ -88,11 +91,8 @@ cd apps/api && uv run alembic revision --autogenerate -m "description"  # New mi
 
 ## Environment
 
-Required `.env` at repo root (see docker-compose.yml):
-- `DATABASE_URL` — Postgres connection string
-- `SECRET_KEY` — app secret
-- `GITHUB_APP_ID`, `GITHUB_WEBHOOK_SECRET` — GitHub App config
-- `FERNET_KEY` — encryption key for stored credentials
+Required `.env` at repo root — see `.env.example` for all variables with generation instructions.
+Key additions for production: `ENVIRONMENT=production`, `ADMIN_PASSWORD`, `CORS_ORIGINS`, `CONTAINER_TTL_SECONDS`, `UVICORN_WORKERS`.
 
 ## Gotchas
 
@@ -113,6 +113,8 @@ Required `.env` at repo root (see docker-compose.yml):
 - **Shallow clone + `gh pr checkout --detach`**: `gh pr checkout` (without `--detach`) fails on `--depth=1` clones because git can't set up tracking branches from shallow refs. Always use `--detach` — containers don't need tracking branches, just files on disk.
 - **Installation IDs in URLs**: frontend routes (`/installations/:id`) use `github_installation_id` (integer, e.g. `123093268`), NOT the internal UUID. The API installation endpoints also expect the GitHub integer ID.
 - **Fast-failing container race**: if a container exits before the SSE stream fully drains, the client may disconnect before `mark_completed()` runs, leaving the session stuck as RUNNING with 0 persisted events. The 5-minute cleanup task marks these as TIMEOUT. Root cause: generator cancellation on client disconnect skips the post-stream `mark_completed` call in `_event_stream()`.
+- **Flaky dispatcher tests**: `test_issues_opened_is_ignored_and_logged` and `test_pull_request_closed_is_ignored` fail when run as part of the full suite due to structlog `configure_logging()` state contamination from `create_app()` in earlier tests. They pass in isolation.
+- **Multi-worker background tasks**: with `--workers N`, each uvicorn worker runs its own lifespan (webhook reaper + container cleanup). Both are idempotent: reaper uses atomic row-level claim (`mark_processing`), cleanup suppresses double-stop exceptions.
 
 ## Key Decisions
 
@@ -122,3 +124,5 @@ Required `.env` at repo root (see docker-compose.yml):
 - **Dashboard over SQLAdmin**: user-facing operations (installation list, session history, token config) go through the dashboard UI; SQLAdmin is the superadmin escape hatch
 - **Open source target**: designed for self-hosting with own Claude licenses
 - **Post-results to PR**: after session completion, the API can post score card as a PR comment — opt-in per installation via `post_results_to_pr` boolean; extraction and formatting in `container/pr_comment.py`, triggered in `_event_stream()` after `mark_completed()`
+- **Coolify deployment**: TLS terminates at the Coolify reverse proxy, not in nginx. Nginx serves the SPA with security headers but no HTTPS config. SSE `X-Accel-Buffering: no` header is set by the API, not nginx.
+- **Non-root API container**: production Dockerfile uses `appuser`. Port 8000 > 1024 so no privilege needed. Docker socket mount still grants Docker access regardless of USER.

apps/api/src/helprs/core/config.py

@@ -2,7 +2,7 @@
 
 from functools import lru_cache
 
-from pydantic import field_validator
+from pydantic import field_validator, model_validator
 from pydantic_settings import BaseSettings, SettingsConfigDict
 
 
@@ -42,6 +42,9 @@ class Settings(BaseSettings):
     # string is for link composition. See deferred-work.md nice-to-have #11.
     APP_BASE_URL: str = "http://localhost:5173"
 
+    # Container orchestration
+    CONTAINER_TTL_SECONDS: int = 900  # 15 minutes
+
     # Environment
     ENVIRONMENT: str = "development"
 
@@ -61,6 +64,28 @@ def validate_fernet_key(cls, v: str) -> str:
             ) from e
         return v
 
+    @model_validator(mode="after")
+    def validate_production_secrets(self) -> "Settings":
+        """Enforce that critical secrets are set when ENVIRONMENT is production."""
+        if self.ENVIRONMENT != "production":
+            return self
+        missing: list[str] = []
+        if not self.ADMIN_PASSWORD:
+            missing.append("ADMIN_PASSWORD")
+        if len(self.SECRET_KEY) < 32:
+            missing.append("SECRET_KEY (must be >= 32 characters)")
+        if not self.GITHUB_WEBHOOK_SECRET:
+            missing.append("GITHUB_WEBHOOK_SECRET")
+        if not self.GITHUB_APP_PRIVATE_KEY:
+            missing.append("GITHUB_APP_PRIVATE_KEY")
+        if not self.GITHUB_CLIENT_ID:
+            missing.append("GITHUB_CLIENT_ID")
+        if not self.GITHUB_CLIENT_SECRET:
+            missing.append("GITHUB_CLIENT_SECRET")
+        if missing:
+            raise ValueError(f"Production environment requires: {', '.join(missing)}")
+        return self
+
 
 @lru_cache
 def get_settings() -> Settings:

apps/api/src/helprs/core/middleware.py

@@ -95,8 +95,8 @@ def setup_middleware(app: FastAPI, settings: Settings) -> None:
         CORSMiddleware,
         allow_origins=settings.CORS_ORIGINS,
         allow_credentials=True,
-        allow_methods=["*"],
-        allow_headers=["*"],
+        allow_methods=["GET", "POST", "PUT", "DELETE", "OPTIONS"],
+        allow_headers=["Authorization", "Content-Type", "X-Request-ID", "Last-Event-ID"],
     )
 
     # Request logging

apps/api/src/helprs/main.py

@@ -5,7 +5,8 @@
 from contextlib import asynccontextmanager
 
 import structlog
-from fastapi import APIRouter, FastAPI
+from fastapi import APIRouter, FastAPI, Request
+from fastapi.responses import JSONResponse
 
 from helprs.core.config import get_settings
 from helprs.core.database import (
@@ -113,13 +114,14 @@ async def _run_container_cleanup(
     """
     from helprs.modules.container.service import AioDockerClient, cleanup_expired
 
+    ttl = get_settings().CONTAINER_TTL_SECONDS
     try:
         while True:
             await asyncio.sleep(interval_seconds)
             try:
                 docker = AioDockerClient()
                 async with app.state.session_factory() as db:
-                    cleaned = await cleanup_expired(db, docker)
+                    cleaned = await cleanup_expired(db, docker, ttl_seconds=ttl)
                     await db.commit()
                     if cleaned:
                         logger.info("container_cleanup_cycle", cleaned=cleaned)
@@ -164,6 +166,18 @@ async def lifespan(app: FastAPI):
             # server starts serving traffic immediately (AC #2).
             await _replay_pending_webhook_events(app)
 
+            # Reconcile container sessions left in RUNNING/PENDING by a
+            # prior crash — mark them FAILED immediately rather than
+            # waiting for TTL expiry.
+            try:
+                from helprs.modules.container.service import reconcile_stale_sessions
+
+                async with session_factory() as db:
+                    await reconcile_stale_sessions(db)
+                    await db.commit()
+            except Exception:
+                logger.exception("session_reconciliation_failed")
+
             # Periodic reaper: handles rows that get stuck mid-run (e.g.
             # mark_processed commit failure) without waiting for the next
             # restart.
@@ -188,6 +202,20 @@ async def lifespan(app: FastAPI):
             if tracked:
                 await asyncio.gather(*tracked, return_exceptions=True)
 
+            # Stop all running containers before shutting down.
+            try:
+                from helprs.modules.container.service import AioDockerClient, cleanup_all_running
+
+                docker = AioDockerClient()
+                async with session_factory() as db:
+                    stopped = await cleanup_all_running(db, docker)
+                    await db.commit()
+                    if stopped:
+                        logger.info("shutdown_containers_stopped", count=stopped)
+                await docker.close()
+            except Exception:
+                logger.exception("shutdown_container_cleanup_failed")
+
             clear_session_factory()
             await engine.dispose()
 
@@ -201,6 +229,15 @@ async def lifespan(app: FastAPI):
     # Exception handlers
     app.add_exception_handler(DomainError, domain_exception_handler)
 
+    async def _unhandled_exception_handler(request: Request, exc: Exception) -> JSONResponse:
+        logger.exception("unhandled_exception", path=request.url.path)
+        return JSONResponse(
+            status_code=500,
+            content={"error": "internal_error", "message": "An unexpected error occurred"},
+        )
+
+    app.add_exception_handler(Exception, _unhandled_exception_handler)
+
     # Middleware (CORS, logging, rate limiting)
     setup_middleware(app, settings)
 
@@ -222,7 +259,17 @@ async def lifespan(app: FastAPI):
     # Health check
     @app.get("/health")
     async def health_check():
-        return {"status": "ok"}
+        from sqlalchemy import text
+
+        try:
+            async with app.state.session_factory() as db:
+                await db.execute(text("SELECT 1"))
+            return {"status": "ok", "db": "ok"}
+        except Exception:
+            return JSONResponse(
+                status_code=503,
+                content={"status": "degraded", "db": "unreachable"},
+            )
 
     return app
 

apps/api/src/helprs/modules/container/router.py

@@ -3,11 +3,11 @@
 import json
 from uuid import UUID
 
-from fastapi import APIRouter, Request
+from fastapi import APIRouter, Depends, Request
 from fastapi.responses import StreamingResponse
 
 from helprs.core.database import get_db_context
-from helprs.core.dependencies import DbSession, GetSettings
+from helprs.core.dependencies import DbSession, GetSettings, get_current_user
 from helprs.core.exceptions import NotFoundError
 from helprs.core.middleware import limiter
 from helprs.core.security import fernet_decrypt
@@ -49,6 +49,7 @@ async def create_container_session(
     request: Request,
     db: DbSession,
     settings: GetSettings,
+    user=Depends(get_current_user),  # noqa: B008
 ):
     """Create a container session and start the container.
 
@@ -87,6 +88,7 @@ async def create_container_session(
         pr_number=body.pr_number,
         repo_full_name=body.repo_full_name,
         skill_name=body.skill_name,
+        user_id=user.id,
     )
 
     # Start the container
@@ -112,6 +114,7 @@ async def get_container_session(
     session_id: UUID,
     request: Request,
     db: DbSession,
+    user=Depends(get_current_user),  # noqa: B008
 ):
     """Get the current status of a container session."""
     cs = await get_session_or_404(db, session_id)
@@ -125,6 +128,7 @@ async def stream_container_output(
     session_id: UUID,
     request: Request,
     db: DbSession,
+    user=Depends(get_current_user),  # noqa: B008
     offset: int = 0,
 ):
     """SSE endpoint streaming container stdout/stderr.
@@ -189,6 +193,7 @@ async def get_session_events_endpoint(
     session_id: UUID,
     request: Request,
     db: DbSession,
+    user=Depends(get_current_user),  # noqa: B008
 ):
     """Retrieve persisted stream-json events for a session.
 
@@ -210,6 +215,7 @@ async def send_session_message(
     body: SendMessageRequest,
     request: Request,
     db: DbSession,
+    user=Depends(get_current_user),  # noqa: B008
 ):
     """Send a user message to a running container session.
 
@@ -235,6 +241,7 @@ async def stop_container_session(
     session_id: UUID,
     request: Request,
     db: DbSession,
+    user=Depends(get_current_user),  # noqa: B008
 ):
     """Stop a running container session."""
     docker = _get_docker_client()

apps/api/src/helprs/modules/container/service.py

@@ -34,8 +34,8 @@
 
 logger = structlog.get_logger()
 
-# Container TTL: 15 minutes maximum
-CONTAINER_TTL_SECONDS = 15 * 60
+# Default container TTL (overridden by Settings.CONTAINER_TTL_SECONDS)
+_DEFAULT_CONTAINER_TTL_SECONDS = 15 * 60
 
 # Base image for the claude-runner container
 CLAUDE_RUNNER_IMAGE = "helprs/claude-runner:latest"
@@ -546,6 +546,7 @@ async def mark_completed(
     db: AsyncSession,
     session_id: UUID,
     docker: DockerClient,
+    ttl_seconds: int = _DEFAULT_CONTAINER_TTL_SECONDS,
 ) -> ContainerSession:
     """Wait for the container to finish, capture exit code, clean up."""
     cs = await get_session_or_404(db, session_id)
@@ -556,7 +557,7 @@ async def mark_completed(
     try:
         exit_code = await asyncio.wait_for(
             docker.wait_container(cs.container_id),
-            timeout=CONTAINER_TTL_SECONDS,
+            timeout=ttl_seconds,
         )
         cs.status = ContainerStatus.COMPLETED if exit_code == 0 else ContainerStatus.FAILED
     except TimeoutError:
@@ -580,12 +581,13 @@ async def mark_completed(
 async def cleanup_expired(
     db: AsyncSession,
     docker: DockerClient,
+    ttl_seconds: int = _DEFAULT_CONTAINER_TTL_SECONDS,
 ) -> int:
     """Find sessions past TTL and destroy their containers.
 
     Returns the number of sessions cleaned up.
     """
-    cutoff = datetime.now(UTC).timestamp() - CONTAINER_TTL_SECONDS
+    cutoff = datetime.now(UTC).timestamp() - ttl_seconds
     cutoff_dt = datetime.fromtimestamp(cutoff, tz=UTC)
 
     result = await db.execute(
@@ -617,3 +619,56 @@ async def cleanup_expired(
         await logger.ainfo("expired_sessions_cleaned", count=cleaned)
 
     return cleaned
+
+
+async def cleanup_all_running(
+    db: AsyncSession,
+    docker: DockerClient,
+) -> int:
+    """Stop ALL running/pending sessions. Used during graceful shutdown."""
+    result = await db.execute(
+        select(ContainerSession).where(
+            ContainerSession.status.in_([ContainerStatus.RUNNING, ContainerStatus.PENDING]),
+        )
+    )
+    sessions = list(result.scalars().all())
+
+    cleaned = 0
+    for cs in sessions:
+        if cs.container_id:
+            with contextlib.suppress(Exception):
+                await docker.stop_container(cs.container_id)
+                await docker.remove_container(cs.container_id, force=True)
+        cs.status = ContainerStatus.COMPLETED
+        cs.completed_at = datetime.now(UTC)
+        cleaned += 1
+
+    if cleaned:
+        await db.flush()
+        await logger.ainfo("shutdown_sessions_stopped", count=cleaned)
+
+    return cleaned
+
+
+async def reconcile_stale_sessions(db: AsyncSession) -> int:
+    """Mark any RUNNING/PENDING sessions as FAILED on boot.
+
+    After a crash, containers are gone but DB records still show RUNNING.
+    This reconciles them immediately rather than waiting for TTL expiry.
+    """
+    result = await db.execute(
+        select(ContainerSession).where(
+            ContainerSession.status.in_([ContainerStatus.RUNNING, ContainerStatus.PENDING]),
+        )
+    )
+    stale = list(result.scalars().all())
+
+    for cs in stale:
+        cs.status = ContainerStatus.FAILED
+        cs.completed_at = datetime.now(UTC)
+
+    if stale:
+        await db.flush()
+        await logger.ainfo("stale_sessions_reconciled", count=len(stale))
+
+    return len(stale)