health-assistant-io
diff --git a/‎CHANGELOG.md‎
Lines changed: 2 additions & 0 deletions b/‎CHANGELOG.md‎
Lines changed: 2 additions & 0 deletions
diff --git a/‎backend/alembic/versions/s1t2m3o4d5e6_add_ai_model_type.py‎
Lines changed: 51 additions & 0 deletions b/‎backend/alembic/versions/s1t2m3o4d5e6_add_ai_model_type.py‎
Lines changed: 51 additions & 0 deletions
diff --git a/‎backend/app/ai/assistance/stt.py‎
Lines changed: 154 additions & 0 deletions b/‎backend/app/ai/assistance/stt.py‎
Lines changed: 154 additions & 0 deletions
diff --git a/‎backend/app/ai/providers/capabilities.py‎
Lines changed: 69 additions & 0 deletions b/‎backend/app/ai/providers/capabilities.py‎
Lines changed: 69 additions & 0 deletions
diff --git a/‎backend/app/ai/providers/enums.py‎
Lines changed: 1 addition & 0 deletions b/‎backend/app/ai/providers/enums.py‎
Lines changed: 1 addition & 0 deletions
@@ -12,6 +12,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+- **AI chat: voice input (speech-to-text) + capability-based model features.** Two coupled changes that rework how models declare what they can do and add hands-free chat input. **(1) Capability-based models:** replaced the implicit "every model is a chat model" assumption with an explicit capability set. A new `AIModelCapability` enum (`text` / `vision` / `audio_input`) is stored as a JSONB `capabilities` array on `AIModel` (migration `s1t2m3o4d5e6`, default `["text"]` so existing models keep their meaning). A single source-of-truth mapping (`app/ai/providers/capabilities.py` `required_capabilities_for_task`) declares what each task needs — `chat`→text, `ocr`→vision, `transcription`→audio_input, every other text-generation task→text — so the AI-config Models UI now shows **feature toggles** (Text is always on; Vision / Audio Input are opt-in chips with icons + colored badges on the list row). A model can be multimodal (e.g. `gpt-4o` = text+vision) or purpose-built (`whisper-1` = audio_input). The `transcription` task type was added to `TaskType`, surfaced in the config summary (`AIConfigSummary.transcription`) + a new `voice_input` workflow, so admins can create a `transcription` task assignment pointing at an `audio_input` model. **(2) Voice input:** a new `POST /ai-assistance/transcribe` endpoint accepts a compressed audio upload, resolves the `transcription` assignment via a new `AIProviderService.get_stt_target` (validates the resolved model advertises `audio_input`; rejects a misconfigured chat-only assignment with a clear guard message), and POSTs it to the provider's `/audio/transcriptions` (OpenAI-compatible) returning `{text}`. Audio is **ephemeral** — never persisted (it may contain PHI); the resulting text is prompt-guard-scanned for audit correlation and then placed in the chat input box for the user to review/edit before sending (it then flows through the normal chat pipeline). Size (20 MiB) + MIME guards at the trust boundary; backend `app/ai/assistance/stt.py` (`STTTarget`, `TranscriptionError`, `transcribe_audio`) + new config (`AI_STT_MAX_AUDIO_BYTES`, `AI_STT_TIMEOUT_SECONDS`, `OPENAI_STT_MODEL`). **Frontend:** `useVoiceRecorder` hook records Opus/16 kHz mono/24 kbps via `MediaRecorder` (~90 KB / 30s — tiny uploads) with a 60s hard cap; `ChatVoiceButton` is a **single unified control** handling both interaction models by gesture — tap (down→up < 250 ms) = toggle (tap to start, tap to stop), press & hold = push-to-talk — with a live elapsed timer and a clear "unsupported/permission-denied" path. Transcribed text appends into the editable input box (preserves typed draft + focuses for review). i18n added (en + el). 12 backend tests (`test_stt_capabilities.py`) cover the capability mapping, baseline inclusion, and STT-target resolution/enforcement; the squash-schema test was relaxed to allow a baseline + incremental follow-ups. Frontend tsc + eslint + build green.
+
 - **AI chat: multimodal image support — ask questions with images.** The agentic chatbot (both the side-panel drawer and the full-screen `/ai-assistant` page) now accepts image attachments so users can ask about lab-report scans, photos, charts, or medication labels. **Backend:** a new modular `app/ai/assistance/attachments.py` is the single source of truth for image handling — an `AllowedImageMime` enum (JPEG/PNG/WEBP/GIF; SVG deliberately excluded as a script vector), `validate_image_data_url` / `validate_chat_images` (MIME whitelist + per-image `AI_CHAT_MAX_IMAGE_BYTES` 8 MiB + per-request `AI_CHAT_MAX_IMAGES` 4 count enforcement at the trust boundary, fail-fast `ImageValidationError` subclassing `ValueError` so it surfaces as a localized guard SSE message with no SDK leak), `build_multimodal_content` (emits the OpenAI vision content-block schema — plain `str` when no images, `[{"type":"text"},{"type":"image_url"}]` otherwise), and `has_images`. `AIAssistanceRequest.images: List[str]` (RFC 2397 data URLs) flows through the `/assist` + `/stream` endpoints; `AIAssistanceService.assist` validates images for the `chat` task only and threads them through `_chat_stream` / `_general_chat`. User messages now persist `{"text", "images": [...]}` in the `ChatMessage.content` JSONB, and `reconstruct_history` rebuilds multimodal `HumanMessage`s for past turns so vision context survives across turns. The chat system prompt gained an "IMAGE & VISION INPUT" rule block (examine images, transcribe values, never fabricate illegible numbers, route "look at my uploaded document" to `get_document_content`). Image bytes bypass the text prompt-injection guard (the HITL wall remains the structural defence for clinical writes). **Frontend:** modular, reusable pieces — `types/ai.ts` gained `ChatImageAttachment` (data URL), a `ChatAttachmentStatus` enum + `PendingChatAttachment` interface for composer state, and `images?` on `Message`; a new `useChatAttachments` hook centralizes client-side validation (mirrors the backend limits so rejected files are never base64-encoded) + `FileReader` data-URL encoding for three input vectors (pick, drag-drop onto the input bar, clipboard paste); `ChatAttachmentPicker` (paperclip button + hidden `<input type=file multiple>`) and `ChatAttachmentPreviewRail` (thumbnail chips with remove + encoding spinner) compose into the modernized input bar; `ChatMessageImages` renders galleries inside bubbles with count-adaptive layouts (single large tile / side-by-side / 1+2 / 2×2 grid with "+N" overflow) and a keyboard-navigable full-screen lightbox. The send button enables image-only sends ("what's this?"), and history reload maps persisted `images` back into the bubbles. 13 backend unit tests (`test_chat_attachments.py`); frontend tsc + eslint + vite build green.
 
 - **Frontend: catalog list rows are now fully clickable + anatomy form unified with the Anatomy Explorer.** Three coupled UX/consistency improvements to the Catalogs workspace and the Anatomy page. (1) In `CatalogBrowser`, each list row and card was selectable only via the inner title/description button — the rest of the row (empty space, the badges column) did nothing. Selection moved to the whole `<li>`/card (cursor-pointer), while the inner interactive elements (class/scope badges, the external-domain link, the picker Add button) keep working independently via `stopPropagation`. (2) Catalogs > Anatomy's create/edit form previously fell back to the bare `GenericCatalogForm` (name + description only), unlike the Anatomy Explorer's `AnatomyStructureForm` (name, slug, category, SNOMED code, description). A new **`AnatomyForm`** is registered in the `CATALOG_FORMS` registry and is now the single shared field set used by **both** surfaces — `AnatomyStructureForm` was refactored to render it. It matches the patterns of the other catalog forms: the anatomy **class** is a `CatalogItemPicker` (`conceptKind="anatomy_class"`, single-select, bound to `class_concept_id`) instead of a legacy uppercase-enum dropdown, and **coding system + code** are two separate fields (`standard_system` select + `standard_code` text) like ConceptForm/BiomarkerForm. (3) Added an **"Open in catalog"** link from both the Anatomy detail card (`/catalogs?type=anatomy&item=<id>`) and the Explorer toolbar (`/catalogs?type=anatomy`) so a structure can be opened/edited in the full catalog workspace. Writes for anatomy now route through the `/anatomy` domain endpoint (not `/catalogs/anatomy`) via a new anatomy entry in the `writeTarget.ts` dispatcher (mirroring how `concept` writes go through `/concepts`), so the class concept is resolved + global-item RBAC enforced by the anatomy service; `buildWritePayload` gained an `anatomy` branch. Dropped the legacy `category` field from the `anatomyService` TS types (`AnatomyListParams`/`AnatomyStructureInput`/`AnatomyStructurePatch`) and the `list()` method — the modern field is `class_concept_id`/`class_concept_slug`; `AnatomySearchPopup` now converts its category filter to a class slug at the API boundary, and `bodyPartService` creates with `class_concept_slug: 'other'`. Frontend tsc + eslint clean.
 
@@ -0,0 +1,51 @@
+"""add ai_models.capabilities (text/vision/audio_input)
+
+Adds a ``capabilities`` JSONB array to ``ai_models`` declaring which
+modalities a model supports (its "features"): ``text`` (baseline, every model),
+``vision`` (image input — multimodal chat / vision OCR), ``audio_input``
+(speech-to-text — the ``transcription`` task). Tasks require specific
+capabilities so the task-assignment picker only offers eligible models.
+
+Defaults to ``["text"]`` so every pre-existing model keeps its current meaning
+(a plain chat/LLM model). The new ``transcription`` task assignment resolves
+to models that advertise ``audio_input`` (e.g. ``whisper-1``).
+
+Revision ID: s1t2m3o4d5e6
+Revises: 8ddb7ef7ca4d
+Create Date: 2026-07-16
+"""
+
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.dialects import postgresql
+
+revision = "s1t2m3o4d5e6"
+down_revision = "8ddb7ef7ca4d"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "ai_models",
+        sa.Column(
+            "capabilities",
+            postgresql.JSONB(astext_type=sa.Text()),
+            nullable=False,
+            server_default=sa.text("'[\"text\"]'"),
+        ),
+    )
+    # GIN index supports capability-containment lookups
+    # (``capabilities ? 'vision'`` / ``capabilities @> '["audio_input"]'``).
+    op.create_index(
+        "ix_ai_models_capabilities",
+        "ai_models",
+        ["capabilities"],
+        unique=False,
+        postgresql_using="gin",
+    )
+
+
+def downgrade() -> None:
+    op.drop_index("ix_ai_models_capabilities", table_name="ai_models")
+    op.drop_column("ai_models", "capabilities")
@@ -0,0 +1,154 @@
+"""Speech-to-text transcription against an OpenAI-compatible API.
+
+STT is a different endpoint from chat completions (``POST /audio/transcriptions``,
+multipart form), so the LangChain ``ChatOpenAI`` chat factory cannot serve it.
+This module is the thin client that resolves the ``transcription`` task
+assignment (a model advertising the ``audio_input`` capability) and POSTs the
+compressed audio to the provider, returning plain text.
+
+Security / privacy
+------------------
+Audio may contain PHI. It is transcribed and then **discarded** — never written
+to the DB, never logged at payload level. The resulting text flows through the
+existing chat pipeline (prompt guard + HITL wall) once it becomes ``user_input``.
+
+Performance
+-----------
+Callers should send Opus/WebM at ~16 kHz mono (~24 kbps) — already tiny. The
+``AI_STT_MAX_AUDIO_BYTES`` guard rejects anything oversized before the network
+call. The whole operation is batch (one round-trip), not streamed.
+"""
+
+from __future__ import annotations
+
+import logging
+from dataclasses import dataclass
+from typing import Optional, Tuple
+
+import httpx
+
+from app.ai.providers.capabilities import required_capabilities_for_task
+from app.ai.providers.enums import TaskType
+from app.core.config import settings
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class STTTarget:
+    """Resolved STT provider/model to call + the auth/base URL."""
+
+    api_key: Optional[str]
+    api_base: str
+    model_name: str
+
+
+class TranscriptionError(RuntimeError):
+    """Raised when transcription is unavailable or the provider call fails.
+
+    Subclasses ``RuntimeError`` (not ``ValueError``) so it is NOT mistaken for
+    a soft guard message by the endpoint error classifier — it surfaces as a
+    generic streaming/HTTP error with no SDK text leak.
+    """
+
+
+def _resolve_stt_target(provider, model) -> STTTarget:
+    """Build the STT call target from a resolved provider+model (or env fallback).
+
+    Validates the resolved model advertises the ``audio_input`` capability —
+    a misconfigured assignment to a chat-only model is rejected early with a
+    clear message instead of a cryptic provider 400.
+    """
+    api_key = provider.get_api_key_plaintext() if provider else None
+    api_base = (
+        provider.api_base if provider and provider.api_base else "https://api.openai.com/v1"
+    )
+    model_name = model.model_name if model else settings.OPENAI_STT_MODEL
+
+    required = required_capabilities_for_task(TaskType.TRANSCRIPTION.value)
+    caps = model.get_capabilities() if model and hasattr(model, "get_capabilities") else None
+    have = {str(c) for c in caps} if caps else set()
+    if required and not any(c.value in have for c in required):
+        raise TranscriptionError(
+            f"Configured STT model '{model_name}' does not advertise the "
+            f"'audio_input' capability. Assign a speech-to-text model "
+            f"(e.g. whisper-1) to the transcription task."
+        )
+
+    if not api_key:
+        raise TranscriptionError(
+            "No API key configured for speech-to-text. Set an OPENAI_API_KEY "
+            "or assign a transcription provider with a key."
+        )
+
+    return STTTarget(api_key=api_key, api_base=api_base.rstrip("/"), model_name=model_name)
+
+
+async def transcribe_audio(
+    audio_bytes: bytes,
+    *,
+    filename: str,
+    mime_type: str,
+    target: STTTarget,
+) -> str:
+    """POST compressed audio to ``{api_base}/audio/transcriptions`` and return
+    the transcribed text.
+
+    The audio is sent as multipart form data (``file`` + ``model``) per the
+    OpenAI-compatible API contract. Raises :class:`TranscriptionError` on any
+    failure (timeout, auth, non-2xx). The caller never sees raw provider text
+    beyond the transcribed payload.
+    """
+    base = target.api_base.rstrip("/")
+    url = f"{base}/audio/transcriptions"
+
+    # httpx builds the multipart boundary itself; do NOT pre-set Content-Type.
+    files = {"file": (filename, audio_bytes, mime_type)}
+    data = {"model": target.model_name}
+
+    try:
+        async with httpx.AsyncClient(timeout=settings.AI_STT_TIMEOUT_SECONDS) as client:
+            resp = await client.post(
+                url,
+                headers={"Authorization": f"Bearer {target.api_key}"},
+                files=files,
+                data=data,
+            )
+    except httpx.TimeoutException as exc:
+        raise TranscriptionError("Speech-to-text request timed out.") from exc
+    except httpx.HTTPError as exc:
+        raise TranscriptionError("Speech-to-text service is unreachable.") from exc
+
+    if resp.status_code >= 400:
+        logger.warning(
+            "STT provider returned HTTP %s for model=%s", resp.status_code, target.model_name
+        )
+        if resp.status_code in (401, 403):
+            raise TranscriptionError("Speech-to-text authentication failed.")
+        if resp.status_code == 429:
+            raise TranscriptionError("Speech-to-text rate limit reached. Try again shortly.")
+        raise TranscriptionError(
+            f"Speech-to-text failed (HTTP {resp.status_code})."
+        )
+
+    try:
+        payload = resp.json()
+    except ValueError as exc:
+        raise TranscriptionError("Speech-to-text returned a malformed response.") from exc
+
+    # OpenAI returns {"text": "..."}; tolerate alternate shapes.
+    text = payload.get("text") if isinstance(payload, dict) else None
+    if not text and isinstance(payload, dict):
+        # Some providers nest under "result" or return a bare string.
+        text = payload.get("result") or payload.get("transcript")
+    if not text:
+        raise TranscriptionError("Speech-to-text returned no text.")
+    return str(text).strip()
+
+
+def split_filename(filename: str) -> Tuple[str, str]:
+    """Return ``(stem, ext)`` for a filename, lowercased extension with dot."""
+    if "." in filename:
+        stem, ext = filename.rsplit(".", 1)
+        return stem, f".{ext.lower()}"
+    return filename, ""
@@ -0,0 +1,69 @@
+"""Map AI tasks to the model capabilities they require.
+
+A model advertises a SET of capabilities (``AIModelCapability``: text / vision
+/ audio_input) on its JSONB ``capabilities`` column. Each task type needs at
+least one specific capability; this module is the single source of truth for
+that mapping so the task-assignment picker only offers eligible models and the
+runtime factories can sanity-check their resolved model.
+
+Examples:
+  * ``chat``        → needs ``text``
+  * ``ocr``         → needs ``vision``
+  * ``transcription`` → needs ``audio_input``
+  * every other text-generation task (define_*, magic_fill, …) → ``text``
+"""
+
+from __future__ import annotations
+
+from typing import Iterable, Optional, Set
+
+from app.ai.providers.enums import TaskType
+from app.models.enums import AIModelCapability
+
+
+# The capability each task type REQUIRES (a model must advertise it to be
+# eligible). Tasks not listed default to {TEXT} (the baseline modality every
+# model carries). Kept as TaskType keys so renames are caught at import time.
+TASK_REQUIRED_CAPABILITY: dict = {
+    TaskType.CHAT: {AIModelCapability.TEXT},
+    TaskType.OCR: {AIModelCapability.VISION},
+    TaskType.TRANSCRIPTION: {AIModelCapability.AUDIO_INPUT},
+}
+
+
+def required_capabilities_for_task(task_type: object) -> Set[AIModelCapability]:
+    """Return the set of capabilities a model must have to serve ``task_type``.
+
+    Unknown/unmapped task types (the long tail of text-generation tasks) fall
+    back to ``{TEXT}``.
+    """
+    key = TaskType.from_string(str(task_type)) if task_type is not None else None
+    return set(TASK_REQUIRED_CAPABILITY.get(key, {AIModelCapability.TEXT}))
+
+
+def normalize_capabilities(values: Optional[Iterable[object]]) -> Set[str]:
+    """Coerce a raw capabilities payload (JSONB list of strings/enum) into a
+    clean set of lowercase capability strings.
+
+    ``text`` is the default modality (an empty/null payload falls back to
+    ``{"text"}``) but is NOT forced when the payload already declares other
+    capabilities — an STT-only model like ``whisper-1`` legitimately carries
+    only ``audio_input``.
+    """
+    if not values:
+        return {AIModelCapability.TEXT.value}
+    result: Set[str] = set()
+    for v in values:
+        cap = AIModelCapability.from_string(str(v))
+        if cap is not None:
+            result.add(cap.value)
+    # An explicitly-empty list (e.g. all values were invalid) → text baseline.
+    return result or {AIModelCapability.TEXT.value}
+
+
+def model_supports(
+    capabilities: Optional[Iterable[object]], required: Iterable[AIModelCapability]
+) -> bool:
+    """True when a model's capability set covers ALL ``required`` capabilities."""
+    have = normalize_capabilities(capabilities)
+    return all(c.value in have for c in required)
@@ -95,6 +95,7 @@ class TaskType(str, enum.Enum):
     SUGGEST_CATEGORY_ICON = "suggest_category_icon"
     GENERATE_CATEGORY_ICON = "generate_category_icon"
     CHAT = "chat"
+    TRANSCRIPTION = "transcription"
 
     @classmethod
     def all_values(cls) -> List[str]: