Skip to content

Commit cc51b5f

Browse files
authored
Implement translation storage refactor (#1142)
* feat(api): add ranking item foundation Add method-neutral ranking item storage, content rows, external source rows, and translated-content storage needed for future ranking item translation. Runtime MaxDiff item paths now use ranking item tables internally while public MaxDiff route naming remains temporarily. Add source and translated plaintext storage/backfill support for rich text. Frontend-controlled rich text sends plaintext alongside HTML where wired, while translated plaintext is derived server-side from sanitized translated HTML and kept storage-only. Add explicit Flyway enum split for ranking_item translation source kind, generated schema artifacts, Python generated models, and a one-off TypeScript plaintext backfill script. Update shared HTML-to-text helpers to decode entities consistently and add focused tests. Deploy: agora, api, content-translation-worker, scoring-worker * fix(project): isolate activity card controls Refactor project activity cards so the card navigation is handled by an overlay SpaLink instead of wrapping the entire card content. This keeps the translation toggle as a separate interactive button, preventing its press state from activating the card push effect or triggering card hover/navigation semantics. The visual CTA remains non-interactive, while the overlay link carries the accessible navigation label. Deploy: agora * fix(agora): normalize metadata font weight Replace stale font-weight token references with the canonical normal-weight token so shared metadata lines do not inherit bold parent styles. This keeps timestamps and edited labels visually consistent across conversation and project metadata surfaces. Deploy: agora * fix(agora): unify project contact details Extract the project details sidebar into a shared component used by project and project-conversation pages. This keeps attribution and contact rendering consistent across desktop layouts, reuses the same details content in the mobile project details sheet, and fixes contact-only/mobile cases so the contact section remains reachable without reserving an empty logo column. Also tightens the contact/attribution language prop types and hardens contact card layout for empty subtitles and long names. Deploy: agora * fix(conversation): redirect importing fetches Return an explicit importing state from /conversation/get when the authenticated owner opens an importing conversation directly, and keep anonymous/non-owner callers seeing the same not-found behavior as inaccessible conversations. The frontend now parses the shared DTO response union, redirects importing responses to the existing import status route, and keeps direct conversation cache updates compatible with the ready-response shape. Metadata lookups also exclude importing conversations so action helpers do not operate on them. Deploy: agora, api * fix(agora): block prioritization exports Add an Open in Agora action to project conversation menus and keep external-link trailing icons aligned with the regular action text color. Hide export entry points for prioritization conversations, redirect direct export-page visits once the conversation metadata shows an unsupported ranking conversation, and avoid fetching export readiness/history for unsupported pages. Reject unsupported prioritization export requests in the API with a typed unsupported_conversation_type business failure, and guard readiness/history direct calls from reporting export availability for ranking conversations. Deploy: agora, api * fix(project): show invitation-only activities Include unindexed project conversations in project activity lists while keeping public conversations ordered first. Make invitation-only activity cards non-interactive and omit their conversation slug IDs from the project activity DTO. Add dev project-page access scenarios for public, mixed, and invitation-only activity lists. Deploy: agora, api * fix(project): unify display content Refactor project page and project-context content resolution around a single displayContent payload instead of sending original and translated variants together. Project content now uses sourceVersion backed by project_content.public_id, and manual project localizations are treated as original-only content for that language without translation controls. Changes: - Add project_content.public_id and regenerated migration 80 - Add shared display-content helpers and project display-content DTOs - Add /project/content/fetch for fetching original or translated project content on demand - Update project page, project conversation shell, project activities, and normal conversation project-context pill to consume displayContent/sourceVersion - Align project translation SSE cache updates with the new project content fetch/cache path - Add pure backend resolver tests for source, manual localization, machine translation, missing translation, stale translation, and translation-status lookup scenarios - Sync shared TypeScript and generated Python/OpenAPI artifacts Deploy: agora, api, content-translation-worker * feat(content): implement translation storage refactor Implement the storage and API refactor described in AGENTS_TRANSLATION_NOTES: ranking item display content, dedicated conversation config/import metadata tables, persisted BWS ranking mode, and strict rich-text/plain-text backfills. Changes: - Move import metadata to conversation_import_source at runtime - Move Polis/ranking config reads and writes to dedicated config tables - Rename persisted ranking mode value from maxdiff to bws - Route public BWS endpoints under /ranking/bws - Add ranking item display content and translation handling - Add destructive cleanup migration for legacy MaxDiff item storage - Add explicit rich-text/plain-text operational backfill script - Update API, frontend clients, workers, and generated models - Remove generated OpenAPI Markdown docs from checked-in outputs Deploy: agora, api, import-worker, math-updater, scoring-worker * fix(agora): refresh analysis tab labels Resolve stale subtab labels after changing display language by mapping tab identifiers to translation keys instead of caching translated strings at component setup. This keeps analysis and MaxDiff result tabs reactive to locale changes while preserving type-checked coverage for all known tab identifiers. Deploy: agora * fix(workers): log AI retry lifecycle Add retry-worker-specific Bedrock read timeout settings while keeping their default value at 12 seconds, matching the previous behavior and preserving the math-updater first-pass timeout separately. Add startup and provider-batch lifecycle logs for AI description and translation retry workers so production logs show effective timeout, model, region, concurrency, claim selection, provider elapsed time, and failure timing. Deploy: ai-description-retry-worker, description-translation-retry-worker * fix(workers): tune Bedrock read timeouts Increase math-updater first-pass Bedrock read timeout to 40 seconds so Mistral Large has enough room for normal latency variance while still failing fast relative to async retry processing. Set retry worker Bedrock read timeout defaults and production compose overrides to 60 seconds for out-of-band backlog cleanup. Deploy: math-updater, ai-description-retry-worker, description-translation-retry-worker * fix(api): harden resource access checks Tighten API resource validation across conversation subresources and generated shared schemas. Validate MaxDiff comparison invariants at the shared DTO boundary, scope import/export and live-resource reads consistently, and keep generated clients in sync.\n\nDeploy: api, agora * fix(workers): add AI retry cooldown Add last_error_at tracking to AI description lineage and translation work so retryable provider failures can be held out of claim scans for a configurable cooldown window. The retry workers now pass retry_cooldown_seconds through claim discovery and claiming, log the configured cooldown, and re-scan the primary database after materialization so newly materialized work still goes through the same claimable/cooldown filters before processing. Retryable and non-retryable AI description error codes are centralized, retryable failures record last_error_at, successful/non-processable cleanup clears it, and non-retryable failures remain controlled by the AI description epoch rather than the cooldown. Regenerate the branch-local V0079/V0080 migration artifacts with last_error_at columns, existing error backfill, maxdiff-to-ranking enum rewrite, and deferred plaintext check validation. Deploy: api, ai-description-retry-worker, description-translation-retry-worker * fix(api): avoid replica-lag rereads Refactor post-write flows to carry transaction-local data and update frontend caches without immediate replica-routed refetches. Changes: - Return known eager translation work IDs from conversation/project writes - Build notification SSE payloads from inserted/known rows instead of rereads - Seed export, comment, and survey frontend caches from mutation responses - Return survey gate state for survey config update/delete - Keep analysis primary fallbacks only where freshness is explicitly requested - Document read-after-write guidance Deploy: agora, api, import-worker * refactor(api): split ranking cleanup migrations Split the ranking/maxdiff migration flow into safe schema expansion, validation, and optional destructive cleanup phases.\n\nChanges:\n- Fold the ranking_item content translation enum addition into the regenerated V0079 migration\n- Keep V0079 focused on safe schema expansion and maxdiff-to-ranking enum normalization\n- Keep V0080 focused on post-backfill validation constraints using NOT VALID followed by VALIDATE CONSTRAINT\n- Move destructive legacy maxdiff table and conversation column cleanup into V0081\n- Update Drizzle snapshots and journal for the regenerated migration sequence\n\nDeploy: api * fix(api): split ranking item enum migration Move the content_translation_source_kind ranking_item enum value back into its own Flyway migration so PostgreSQL commits the enum addition before V0079 uses it in partial indexes and check constraints.\n\nThis avoids the unsafe use of new enum value error when Flyway runs V0079 in a transaction.\n\nDeploy: api * fix(api): backfill normalized ranking conversations V0079 now rewrites conversation_type from maxdiff to ranking before V0079.1 runs. Update the ranking config backfill to target the normalized ranking value so PostgreSQL does not reject the removed maxdiff enum value.\n\nDeploy: api
1 parent 2db19b2 commit cc51b5f

1,275 files changed

Lines changed: 82080 additions & 60518 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.gitattributes

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
services/agora/src/api/** whitespace=-blank-at-eol
2+
services/load-testing/src/api/** whitespace=-blank-at-eol

AGENTS.md

Lines changed: 69 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -530,6 +530,17 @@ if (probabilities.length !== types.length) {
530530
- Data from untyped sources (raw SQL, environment variables)
531531
- Legacy code integration where types cannot be guaranteed
532532

533+
### Avoid Replica-Lag Read-After-Write Bugs
534+
535+
The API uses read replicas for ordinary reads, so code that writes data and then immediately reads it back through the normal `db` handle can observe stale state. Prefer designs that avoid the follow-up read entirely:
536+
537+
- Use `.returning()` from inserts/updates for IDs, timestamps, and updated fields
538+
- Build downstream DTOs, queue payloads, notifications, and cache seeds from values already known in the request, transaction, or returned rows
539+
- If a follow-up read is genuinely needed, perform it inside the same write transaction or otherwise ensure it uses a fresh writer-side view
540+
- Do not add primary-read fallbacks as the first solution; use them only when the data cannot reasonably be carried forward or rebuilt from known values
541+
542+
Example: after creating a notification row, prefer broadcasting the SSE payload from the inserted row plus known conversation/opinion data rather than inserting and then rereading the notification through a replica-routed query.
543+
533544
### Prefer `async`/`await` Over `.then()` Chains
534545

535546
Always use `async`/`await` for asynchronous code. Do not use `.then()` or `.catch()` chains. This makes control flow easier to follow, error handling more consistent, and avoids nesting.
@@ -582,6 +593,59 @@ function processImport(files: Partial<Record<string, string>>): void {
582593

583594
**Key principle:** Parsing returns a new value with a more precise type. Validation returns a boolean and requires unsafe casting. Parsing makes invalid states unrepresentable in the type system.
584595

596+
### Use Parsed DTOs as Downstream Types
597+
598+
When an API endpoint has a shared Zod DTO schema with important type-level invariants, parse or safe-parse the request/response at the boundary and consider passing the parsed value downstream. This is especially important for `z.discriminatedUnion()` or other correlated/special shapes that can be weakened or flattened by OpenAPI-generated client types. Do not recreate a parallel TypeScript interface or duplicate those invariants in service/composable props.
599+
600+
Do not apply this mechanically to every DTO. For simple flat request/response shapes where generated OpenAPI types preserve the useful structure, normal explicit parameter objects are fine.
601+
602+
Be careful when changing DTO shapes used by the OpenAPI generator: generated API client types can be temporarily out of sync with the source Zod schemas until `make generate` has run, and some Zod semantics may not round-trip perfectly through OpenAPI. When editing these paths, check both the source DTO schema and generated client/shared copies before relying on either type.
603+
604+
**Preferred approach:**
605+
606+
```typescript
607+
const request = Dto.createNewConversationRequest.parse({
608+
conversationTitle,
609+
conversationType: "ranking",
610+
rankingMode: "maxdiff",
611+
// ...other fields
612+
});
613+
614+
await createNewPost({ db, request });
615+
```
616+
617+
Then downstream code should accept the parsed DTO type:
618+
619+
```typescript
620+
async function createNewPost({
621+
db,
622+
request,
623+
}: {
624+
db: PostgresDatabase;
625+
request: CreateNewConversationRequest;
626+
}) {
627+
if (request.conversationType === "ranking") {
628+
// TypeScript knows request.rankingMode exists here.
629+
useRankingMode(request.rankingMode);
630+
}
631+
}
632+
```
633+
634+
**Avoid:**
635+
636+
```typescript
637+
interface CreateNewPostProps {
638+
conversationType: ConversationType;
639+
rankingMode?: RankingMode;
640+
}
641+
642+
if (conversationType === "ranking" && rankingMode === undefined) {
643+
throw new Error("Missing ranking mode");
644+
}
645+
```
646+
647+
Use `z.discriminatedUnion()` in the DTO schema when fields are correlated, such as `conversationType: "ranking"` requiring `rankingMode`. This makes invalid combinations unrepresentable after parsing and lets TypeScript narrow correctly throughout downstream code.
648+
585649
### Functional Programming Style: Closure Pattern (Zustand-style)
586650

587651
**Preferred Pattern**: Closure-based state encapsulation with immutable API
@@ -746,7 +810,11 @@ In Vue components, prefer **explicit props drilling** over `inject`/`provide` fo
746810

747811
### Vue Props: Prefer Required Nullable Props
748812

749-
For Vue component props, prefer `prop: T | undefined` over `prop?: T`. Optional props can hide typos because a mistyped binding may simply omit the intended prop. A required prop whose value may be `undefined` forces every caller to wire the prop by name while still representing absent data.
813+
For Vue component props, prefer required `prop: T | undefined` over optional `prop?: T`. Optional props can hide typos because a mistyped binding may simply omit the intended prop. A required prop whose value may be `undefined` forces every caller to wire the prop by name while still representing absent data. Callers should pass `some-value-or-undefined` explicitly, not rely on omitted props, when absence is part of the component contract.
814+
815+
### Undefined Over Null in App Code
816+
817+
Prefer `undefined` for absent optional values in frontend and backend application code. In backend service/DAO mappers, normalize database `null` values with `toUnionUndefined()` when the internal or frontend-facing shape represents absence. DTOs sent to the frontend should use `undefined` or omit absent optional fields unless the DTO contract intentionally requires `null`. Use `null` where it is required by the database schema, external APIs, or existing DTO contracts, and convert explicitly at those boundaries.
750818

751819
### Logging Guidelines
752820

0 commit comments

Comments
 (0)