refactor(container): one scorecard source, and a thin session route

Two scorecard extractors coexisted and both ran inside finalize_session: a
validated JSON parser feeding cs.scorecard, and an unvalidated regex over
the markdown feeding the PR comment. Same session, two formats, two failure
modes — a small wording drift in a skill silently produced no comment while
the dashboard scored it fine, and pr_comment.py ran its own SQL over the
events table to do it, in a module whose sibling repository claims to own
every query.

Skills deliberately emit both formats: markdown for the human watching the
stream, JSON for anything that reads the outcome. Only the JSON is parsed
now. The comment is rendered from the same validated object the dashboard
stores, so the two cannot disagree about a session.

The scorecard is a Pydantic model rather than a dict with hand-rolled
checks, which also replaces `REQUIRED_FIELDS`/`len(dims) != 3`/`0 <= v <= 10`
with declarations. `extra="allow"`: the declared fields are the contract,
not the ceiling.

Behaviour change worth knowing: the comment's headline score is now the mean
of the reported dimensions instead of the free-text "Score: 8" the skill
wrote next to them — a number nothing stopped from disagreeing with the
dimensions on the line below it.

xp_earned is finally written, from that same scorecard. It was declared on
the model, created by a migration, exposed in two response schemas and read
by the frontend, and never once assigned.

create_container_session goes from a 45-line handler to 27 lines around one
call. Authorization, Fernet decryption and token minting were transport-
layer work that the webhook path could never reuse; they are `open_session`
in the service now.

Author: mariuspruvot

CLAUDE.md

@@ -143,7 +143,7 @@ Key additions for production: `ENVIRONMENT=production`, `ADMIN_PASSWORD`, `CORS_
 - **BYOK via dashboard**: Claude OAuth tokens (from `claude setup-token`) stored per installation, encrypted with Fernet, injected as `CLAUDE_CODE_OAUTH_TOKEN` env var into containers. Zero API cost (uses user's Claude subscription).
 - **Dashboard over SQLAdmin**: user-facing operations (installation list, session history, token config) go through the dashboard UI; SQLAdmin is the superadmin escape hatch
 - **Open source target**: designed for self-hosting with own Claude licenses
-- **Post-results to PR**: after session completion, the API can post score card as a PR comment — opt-in per installation via `post_results_to_pr` boolean; extraction and formatting in `container/pr_comment.py`, triggered in `_event_stream()` after `mark_completed()`
+- **Post-results to PR**: after session completion the API can post the scorecard as a PR comment — opt-in per installation via `post_results_to_pr`. Rendered from the validated `Scorecard` model (`container/scorecard.py`), the same object stored on `cs.scorecard` and used for `xp_earned`, so the dashboard and the comment cannot disagree. Skills emit two things: markdown for the human watching the stream, and the `helprs-scorecard` JSON block for everything machine-read. Only the JSON is parsed; `pr_comment.py` formats, it no longer extracts.
 - **Coolify deployment**: Two-domain setup via Traefik: `helprs.tech` (web) and `api.helprs.tech` (API). TLS via Let's Encrypt, managed by Coolify. Domains are set in Coolify UI (General > Domains for api / Domains for web) — they may get cleared on redeploy, re-check after each deploy. "Preserve Repository During Deployment" must be enabled so skills are available on the host. The prod compose uses `./` paths (repo-root-relative) because Coolify sets `--project-directory` to the repo root.
 - **claude-runner image**: declared in both `docker-compose.yml` and `infra/coolify/docker-compose.prod.yml` as a **build-only service** — `entrypoint: ["/bin/true"]` + `restart: "no"` so the container exits immediately on `up`, leaving only the built image `claude-runner:latest` on the host. The API spawns containers from this image dynamically via the mounted Docker socket. Image tag is hard-coded in `container/service.py:CLAUDE_RUNNER_IMAGE` — it must stay `claude-runner:latest` (no namespace prefix). Earlier attempt with `profiles: [build-only]` was reverted: `profiles:` excludes a service from both build AND run unless the profile is explicitly activated, which Coolify does not do, causing `404 No such image` failures on session spawn.
 - **Non-root API container**: production Dockerfile uses `appuser` with `chown -R appuser:appuser /app` for uv cache writes. Docker socket access requires `group_add: ["${DOCKER_GID:-994}"]` in the compose to match the host's docker group GID.

apps/api/src/helprs/modules/container/pr_comment.py

@@ -1,68 +1,62 @@
-"""Score card extraction and PR comment formatting for challenge-me sessions."""
+"""Rendering a finished session as a GitHub PR comment.
 
-import re
-from uuid import UUID
+Built from the validated ``Scorecard``, not from the markdown the skill
+writes for the live stream. Two extractors used to coexist -- a validated
+JSON parser feeding the dashboard and an unvalidated regex over the markdown
+feeding this comment -- so one session could be scored in one place and
+silently uncommented in the other. There is one machine-readable source now,
+and this module no longer runs its own SQL over the events table.
+"""
 
-import structlog
-from sqlalchemy import select
-from sqlalchemy.ext.asyncio import AsyncSession
+from uuid import UUID
 
-from helprs.modules.container.models import SessionEvent
+from helprs.modules.container.scorecard import MAX_DIMENSION_SCORE, Scorecard
 
-logger = structlog.get_logger()
+_BAR_WIDTH = 10
 
-_RESULTS_PATTERN = re.compile(
-    r"---\s*\n\s*\n##\s+Results\b(.*?)^---\s*$",
-    re.DOTALL | re.MULTILINE,
-)
 
+def _score_bar(score: float) -> str:
+    filled = round(score * _BAR_WIDTH / MAX_DIMENSION_SCORE)
+    return "█" * filled + "░" * (_BAR_WIDTH - filled)
 
-async def extract_score_card(session_id: UUID, db: AsyncSession) -> str | None:
-    """Extract the score card markdown from persisted session events.
 
-    Queries the last ``result`` event for the session and extracts the
-    ``## Results`` section delimited by ``---`` markers (as defined in
-    ``skills/challenge-me/CLAUDE.md``).
-    """
-    result = await db.execute(
-        select(SessionEvent.data)
-        .where(
-            SessionEvent.session_id == session_id,
-            SessionEvent.data["type"].astext == "result",
-        )
-        .order_by(SessionEvent.event_id.desc())
-        .limit(1)
+def _dimensions_table(scorecard: Scorecard) -> str:
+    rows = "\n".join(
+        f"| {name.replace('_', ' ').title()} | {score:g} / {MAX_DIMENSION_SCORE} |"
+        for name, score in scorecard.dimensions.items()
     )
-    row = result.scalar_one_or_none()
-    if row is None:
-        return None
-
-    result_text = row.get("result")
-    if not result_text or not isinstance(result_text, str):
-        return None
+    return f"| Dimension | Score |\n|-----------|-------|\n{rows}"
 
-    match = _RESULTS_PATTERN.search(result_text)
-    if not match:
-        return None
 
-    return f"## Results{match.group(1).rstrip()}"
+def format_pr_comment(scorecard: Scorecard, session_url: str) -> str:
+    """Render the scorecard as the comment body."""
+    overall = scorecard.overall_score
+    sections = [
+        "### helPRs Challenge-Me Results",
+        "",
+        f"**Score: {overall:.1f} / {MAX_DIMENSION_SCORE}**  {_score_bar(overall)}",
+        "",
+        _dimensions_table(scorecard),
+        "",
+        scorecard.summary,
+    ]
 
+    if scorecard.highlights:
+        sections += ["", "**Highlights**", *(f"- {item}" for item in scorecard.highlights)]
 
-def format_pr_comment(
-    score_card: str,
-    session_url: str,
-) -> str:
-    """Format the score card as a GitHub PR comment."""
-    return (
-        f"### helPRs Challenge-Me Results\n\n"
-        f"{score_card}\n\n"
-        f"<details>\n"
-        f"<summary>View session</summary>\n\n"
-        f"[Open full Q&A session]({session_url})\n\n"
-        f"---\n"
-        f"*Posted by [helPRs](https://github.com/apps/helprs)*\n"
-        f"</details>\n"
-    )
+    sections += [
+        "",
+        "<details>",
+        "<summary>View session</summary>",
+        "",
+        f"[Open full Q&A session]({session_url})",
+        "",
+        "---",
+        "*Posted by [helPRs](https://github.com/apps/helprs)*",
+        "</details>",
+        "",
+    ]
+    return "\n".join(sections)
 
 
 def build_session_url(

apps/api/src/helprs/modules/container/router.py

@@ -14,9 +14,8 @@
 
 from helprs.core.database import get_db_context
 from helprs.core.dependencies import CurrentUser, DbSession, GetSettings, authenticate_token, stream_token
-from helprs.core.exceptions import ConflictError, NotFoundError
+from helprs.core.exceptions import ConflictError
 from helprs.core.middleware import limiter
-from helprs.core.security import fernet_decrypt
 from helprs.modules.container.docker_client import AioDockerClient, DockerClient
 from helprs.modules.container.models import ContainerStatus
 from helprs.modules.container.schemas import (
@@ -30,23 +29,16 @@
     StopSessionResponse,
 )
 from helprs.modules.container.service import (
-    create_session,
     delete_session,
     finalize_session,
     get_session_events,
     get_session_or_404,
+    open_session,
     send_message,
-    start_container,
     stop_container,
 )
 from helprs.modules.container.streaming import spawn_detached, stream_and_persist
-from helprs.modules.installation.github import RUNNER_TOKEN_PERMISSIONS
 from helprs.modules.installation.service import (
-    get_byok_config,
-    get_installation_by_github_id,
-    mint_installation_token,
-    verify_installation_access,
-    verify_repo_access,
     verify_session_access,
 )
 
@@ -70,53 +62,17 @@ async def create_container_session(
     user: CurrentUser,
 ) -> ContainerSessionResponse:
     """Create a session and start its container."""
-    installation = await get_installation_by_github_id(db, body.installation_id)
-    if not installation:
-        raise NotFoundError("Installation not found")
-
-    # Installation membership is not enough: an installation can cover repos
-    # this user cannot read, and the session streams the diff back to them.
-    await verify_installation_access(user, installation, settings)
-    await verify_repo_access(user, body.repo_full_name, settings)
-
-    byok_config = await get_byok_config(db, installation.id)
-    if not byok_config:
-        raise NotFoundError("No Claude token configured for this installation")
-
-    claude_oauth_token = fernet_decrypt(byok_config.encrypted_api_key, settings.fernet_keys)
-
-    # Narrowed to this repo, read-only: the container runs Claude Code over
-    # untrusted PR content with network egress.
-    github_token = await mint_installation_token(
-        installation.github_installation_id,
-        settings,
-        repositories=[body.repo_full_name.split("/")[-1]],
-        permissions=RUNNER_TOKEN_PERMISSIONS,
-    )
-
-    cs = await create_session(
-        db=db,
-        installation_id=installation.id,
-        pr_number=body.pr_number,
-        repo_full_name=body.repo_full_name,
-        skill_name=body.skill_name,
-        user_id=user.id,
-    )
-    # Committed before the container is touched. start_container marks the row
-    # FAILED on error and then raises, but that exception unwinds through
-    # get_db, which rolls back -- discarding the FAILED status *and* the row
-    # itself. A failed start left the user with a 502 and a dashboard showing
-    # that nothing had ever happened.
-    await db.commit()
-
     docker = _get_docker_client()
     try:
-        cs = await start_container(
-            db=db,
-            session_id=cs.id,
-            docker=docker,
-            claude_oauth_token=claude_oauth_token,
-            github_token=github_token,
+        cs = await open_session(
+            db,
+            docker,
+            user=user,
+            installation_github_id=body.installation_id,
+            pr_number=body.pr_number,
+            repo_full_name=body.repo_full_name,
+            skill_name=body.skill_name,
+            settings=settings,
         )
     finally:
         await docker.close()

apps/api/src/helprs/modules/container/scorecard.py

@@ -1,47 +1,74 @@
-"""Scorecard extraction from Claude session output.
+"""The structured scorecard a skill emits at the end of a session.
 
-Skills emit a structured JSON scorecard wrapped in a fenced code block
-tagged `helprs-scorecard`. This module parses it from the last assistant
-message text.
+Skills write two things when they finish: markdown results for the human
+watching the stream, and this JSON block for anything that has to *read* the
+outcome. Both are documented in ``skills/challenge-me/CLAUDE.md``.
+
+Everything machine-consumed goes through the model below, so the dashboard
+and the PR comment can never disagree about a session. The PR comment used to
+be regex-scraped out of the markdown instead, with no validation, so a small
+wording drift in a skill silently produced no comment at all.
 """
 
 import json
 import re
 
+from pydantic import BaseModel, ConfigDict, Field, ValidationError, field_validator
+
 SCORECARD_PATTERN = re.compile(
     r"```helprs-scorecard\s*\n(.*?)\n```",
     re.DOTALL,
 )
 
-REQUIRED_FIELDS = {"skill", "version", "dimensions", "summary"}
+DIMENSION_COUNT = 3
+MAX_DIMENSION_SCORE = 10
 
 
-def extract_scorecard(text: str) -> dict | None:
-    """Extract the helprs-scorecard JSON block from text.
+class Scorecard(BaseModel):
+    """A skill's verdict on one session.
 
-    Returns the parsed dict if found and valid, None otherwise.
+    ``extra="allow"`` because a skill may report more than helPRs reads: the
+    fields below are the contract, not the ceiling.
     """
-    match = SCORECARD_PATTERN.search(text)
-    if not match:
-        return None
 
-    try:
-        data = json.loads(match.group(1))
-    except (json.JSONDecodeError, ValueError):
-        return None
+    model_config = ConfigDict(extra="allow")
 
-    if not isinstance(data, dict):
-        return None
+    skill: str
+    version: int
+    dimensions: dict[str, float]
+    summary: str
+    questions_asked: int | None = None
+    questions_answered: int | None = None
+    highlights: list[str] = Field(default_factory=list)
 
-    if not REQUIRED_FIELDS.issubset(data.keys()):
-        return None
+    @field_validator("dimensions")
+    @classmethod
+    def check_dimensions(cls, value: dict[str, float]) -> dict[str, float]:
+        if len(value) != DIMENSION_COUNT:
+            raise ValueError(f"expected exactly {DIMENSION_COUNT} dimensions, got {len(value)}")
+        for name, score in value.items():
+            if not 0 <= score <= MAX_DIMENSION_SCORE:
+                raise ValueError(f"dimension '{name}' is {score}, outside 0-{MAX_DIMENSION_SCORE}")
+        return value
+
+    @property
+    def overall_score(self) -> float:
+        """Mean of the dimensions, on the same 0-10 scale."""
+        return sum(self.dimensions.values()) / len(self.dimensions)
 
-    dims = data.get("dimensions")
-    if not isinstance(dims, dict) or len(dims) != 3:
-        return None
 
-    for value in dims.values():
-        if not isinstance(value, (int, float)) or value < 0 or value > 10:
-            return None
+def extract_scorecard(text: str) -> Scorecard | None:
+    """Parse the ``helprs-scorecard`` block out of text.
 
-    return data
+    Returns ``None`` rather than raising: a session whose skill emitted no
+    scorecard, or emitted a malformed one, is a session without a scorecard --
+    not a failed session.
+    """
+    match = SCORECARD_PATTERN.search(text)
+    if not match:
+        return None
+
+    try:
+        return Scorecard.model_validate(json.loads(match.group(1)))
+    except (json.JSONDecodeError, ValueError, ValidationError):
+        return None