QA checks 1.0

[ZuzanaOdstrcilova]

1. Problem

Translation quality issues are a common source of bugs and user confusion in localized products. Even with Tolgee's existing features, users still struggle to catch common translation errors such as missing text, inconsistent formatting, broken placeholders, spelling mistakes, punctuation mismatches, and syntax issues. These errors require manual review or external tools, slowing down translation workflows and releases.

Without built-in quality feedback directly in Tolgee's UI, teams spend time hunting for problems that could be automatically detected. Other localization platforms already use automated QA checks to reduce manual effort and improve translation quality.

More of our customers have asked about this feature in the past. At the beginning of last year, we already tested designs with some users, and even though we moved the feature to 2026, it remains a priority on our Roadmap. This feature will make our tool more reliable and valuable for larger companies and those for whom precise, mistake-free translation matters.

The business case / metrics:
We expect that this feature can drive user adoption, customer acquisition and lower churn. It's adds additional layer of reliability above the AI translation.

---

2. Appetite

1 full cycle / 4 weeks
Build the core QA functionality first, then use remaining time to add as many QA checks as possible.

---

3.&4. Solution & UI (basic drawings)

We already have specified solution in Figma designs here:
https://www.figma.com/design/UYsyo57SCcWFpyMirhGmv4/Tasks-2025?node-id=10-14842&t=xZaJw19WPWNPChLa-1

UI Views:

• Dashboard view - show QA issues on each language row (plus dialog with types of QA issues and CTA buttons
• Translation view - icon QA issues, marking of the QA issue in the non-edited text (persisted issues only), live preview QA panel during editing, QA check box on the right side, solving-ignoring of the issue, dialog for quick QA solving, stale-aware loading indicators
• Filtering of QA issues
• Settings of QA
• Support for AI prompt QA check — separate section in QA settings with add/edit/toggle per AI check, reuse AI playground panel for prompt editing (make it generic for both translation and QA modes)

Types of QA:

• AI QA check: Configurable - Important, but cuttable if it becomes a rabbit hole. Basic rule-based checks are higher priority.
• Empty translation
• Lenght issues (out of scope for now, key length functionality is not yet prepared in Tolgee)
• Spaces mismatch (+extra spaces)
• Unmatches new lines
• Character case mismatch
• Missing numbers
• Repeated words
• Punctuation mismatch
• Brackets mismatch
• Special character mismatch
• Different URLs
• Inconsistent placeholders
• Inconsistent HTML
• ICU syntax
• Spelling
• Unresolved comments

5. Rabbit Holes

• Handling the QA checks on backend and propagation to frontend...
  • Do we need to live update QA checks when editing or are we going for async operation?
  • Resolved: Both. Live preview in the QA panel during editing (stateless endpoint). Persisted checks run async on save. A qaChecksStale flag on Translation bridges the gap — set true on save (same transaction), cleared when QA issues are persisted (same transaction). Frontend shows loading state while stale. No inline highlighting during editing — panel is sufficient.
• Handle QA checks in batch when user enables the QA check feature.
• AI QA check - Connection to LLM providers, enabling user to specify custom prompt, enable testing like in AI playground
  • What happens when configuration changes?
  • Clarified: Separate section in QA settings under rule-based checks. Each AI check has add/edit buttons that navigate to AI playground panel (reused). Make AI playground code generic to support both translation and QA check modes. User can set custom prompt, enable/disable system prompt parts, or use advanced mode for fully custom prompt.

6. No-Goes

• Ignoring simple QA checks in favor of complex ones
• Automatic modification of translations
• AI-based linguistic or stylistic evaluation (just connection with user defined AI prompt)

7. Success Criteria

Functional:

• QA issues are visible directly in the translation workflow
• Users can easily identify and fix common translation problems
• Core QA functionality is stable and usable
• A meaningful set of QA checks is implemented
• QA checks do not slow down normal translation work
• Users are satisfied with final translation quality

Business
[Specified in internal Fibery]

Mesurement:

• PostHog events for each QA
• UX Testing with core users
• Internal UX tests

[ZuzanaOdstrcilova]

#2907 - In the conversation below, some specific problems are mentioned.

[Anty0]

Scope Map

Appetite: 4 weeks (1 full cycle) | Developer: 1 | EE feature (gated behind Feature.QA_CHECKS)

Architecture: dual-mode QA execution

QA checks run in two modes, both executing on the backend:

1. Persisted checks (async, on save)

• When a translation is saved, a batch job runs all enabled QA checks
• Results are persisted as TranslationQaIssue entities in the database
• QA issue counts are included in the translations list API response (like commentCount)
• Uses existing BatchJobService infrastructure (BatchJobType.QA_CHECK)

QA checks stale flag

• There is a time gap between a translation being saved and QA checks being updated. A qaChecksStale boolean flag on Translation makes this gap visible to the frontend.
• Set to true synchronously when translation text is modified
• Set to false synchronously when QA issues for that translation are persisted
• Included in TranslationView / TranslationViewModel API response
• Frontend uses this flag to show a loading state on QA badge indicators (see "QA in translation view" and "QA on dashboard" scopes)

2. Live preview (on edit, before save)

• While editing, the frontend calls a lightweight POST /v2/projects/{projectId}/qa-check/preview streaming endpoint
• Sends the draft text + language tag + key ID
• Backend runs enabled checks against the draft text and returns issues (not persisted)
• Pattern follows the machine translation approach, but on top of that reuses connection to get updated issues as user is editing translation text (with some debounce to avoid overloading server)
• Results shown in the QA panel (tools panel section) only — no inline highlighting during editing
• We will probably need to send the issues per QA check type - frontend will then replace all issues related to the check type and sort them by something
• When backend finishes processing all QA checks, it should notify frontend that no more data are incoming, so frontend can hide loading bar at the top of the panel (shown when user starts typing again)

Note: Inline highlights (underlines) in translation text are only shown for persisted QA issues on translations that are not being edited. They do not appear during editing — the live preview panel is sufficient.

QA check result structure

Each check returns issues with:

• Type — QaCheckType enum value
• Message variant enum — a QaIssueMessage enum (similar to Message.kt pattern). Each check type has multiple enum variants for different sub-cases (e.g., SPACES_MISMATCH_LEADING_ADDED, SPACES_MISMATCH_TRAILING_REMOVED). On the frontend, a big switch (like useErrorTranslation.ts) translates each enum value to a localized, actionable user-facing message.
• Replacement text (nullable) — the corrected text (only part of the actual translation, replaces part of translation as defined by position bellow) that should replace the problematic span. null when auto-correction is not possible. The frontend applies this client-side when the user clicks CORRECT (autofill, user must still save manually).
• Position — start/end offsets in the translation text (for inline underlines and to identify what gets replaced)

Heavy check handling (AI QA)

• Rule-based checks are fast (string comparisons, regex) — run in both modes
• AI QA check runs havily debounced in live preview mode — LLM calls are slower and more expensive, so debouncing prevents excessive API usage (this means we need to support different debounce settings for each QA check; ideally frontend sends text changes with small debounce and server then takes care of debouncing the actual QA checks if needed)
• Optional: a manual "Run AI check now" button in the QA panel for on-demand AI checking during editing

---

All individual check scopes are marked ~ (nice-to-have), but we need to implement at least a few — otherwise the feature will look incomplete to the user.

QA check engine

The backend core. New entity, check runner service, dual-mode trigger mechanism.

• New TranslationQaIssue JPA entity (type enum, QaIssueMessage enum for variant-specific message, replacement text (nullable), position start/end, state: open/ignored, reference to Translation)
• New QaCheckType enum for all check types
• New QaCheckSeverity enum: WARNING, OFF (extensible to ERROR in QA 2.0 — use enum from the start but only expose two values)
• QaCheckRunner service with pluggable check interface — each check implements a common interface returning issues with messages, corrections, and positions
• Persisted mode: New BatchJobType.QA_CHECK — runs on translation save, stores issues in DB
• Live preview mode: Raw WebSocket at /ws/qa-preview — persistent connection per editing session, sends results per check type as they complete, supports per-check debounce for future heavy checks (AI QA).
• Add qaIssueCount field to TranslationView and TranslationViewModel (like existing commentCount)
• Add qaChecksStale boolean flag to Translation entity (new column in translation table)
• Set qaChecksStale = true synchronously when translation text is modified (same transaction as save)
• Set qaChecksStale = false synchronously when QA issues are persisted for the translation (same transaction as QA issue save)
• Include qaChecksStale in TranslationView and TranslationViewModel API response
• Repository + query for fetching QA issues per translation
• API endpoint to get detailed persisted QA issues for a specific translation
• I think I know how to do it
• I have validated my approach

Key files to modify/reference:

• backend/data/src/main/kotlin/io/tolgee/constants/Message.kt — pattern for QaIssueMessage enum (328+ values, @JsonValue lowercase codes)
• webapp/src/translationTools/useErrorTranslation.ts — pattern for frontend enum→message switch
• backend/data/src/main/kotlin/io/tolgee/model/translation/Translation.kt — add QA issues relation
• backend/data/src/main/kotlin/io/tolgee/model/views/TranslationView.kt — add QA count
• backend/api/src/main/kotlin/io/tolgee/hateoas/translations/TranslationViewModel.kt — add QA count to API
• backend/data/src/main/kotlin/io/tolgee/batch/data/BatchJobType.kt — add QA_CHECK type
• backend/data/src/main/kotlin/io/tolgee/batch/processors/ — new QA check processor

QA in translation view

Frontend display of QA issues: tools panel section and cell indicators.

• QA section in the tools panel (expandable section alongside MT/TM/Screenshots/Comments/History):
  • Issue count badge in section header (e.g., "QUALITY ASSURANCE 2")
  • Each issue as a mini-section: type name, description (first slice — no old→new diff view yet, added with CORRECT)
  • ✓ (correct) and ✕ (ignore) icon buttons per issue (with tooltips on hover)
  • Show live preview issues during editing
  • Initialize with persisted issues until user starts editing
• QA cell indicators in the translation list:
  • QA warning icon on translation cells
  • QA issue count badge
  • QA badge shows loading when QA issues are stale (the qaChecksStale flag)
  • QA badge automatically updates when new QA issues are generated (should ideally use the same mechanism we already use to keep translations in the translations view up-to-date)
• Stale flag handling on QA indicators:
  • If qaChecksStale is true → show gray QA icon with small loading spinner instead of badge
  • If qaChecksStale is false and has issues → show normal QA icon with badge (issue count)
  • If qaChecksStale is false and no issues → hide QA icon
• useQaCheckPreview hook — calls live preview endpoint with draft text (follows glossary pattern)
• I think I know how to do it
• I have validated my approach

Key files to modify/reference:

• webapp/src/views/projects/translations/ToolsPanel/panelsList.tsx — register QA section
• webapp/src/ee/glossary/hooks/useGlossaryTermHighlights.ts — pattern for live preview hook
• webapp/src/ee/glossary/components/GlossariesPanel.tsx — pattern for tools panel section

QA inline highlighting

Inline underlines on read-only translation text (not during editing) for persisted QA issues from DB. similar to the glossary highlights.

• Underline problematic text spans in TranslationWithPlaceholders using the same modifier/decoration pattern as glossary highlights
• Hover popover on underlined text showing: issue type, description, old→new diff, CORRECT/IGNORE buttons (same component as used in QA panel, but wrapped with card - similar to glossaries)
• Coexist with existing glossary and ICU highlights via sortModifiers()
• I think I know how to do it
• I have validated my approach

Key files to modify/reference:

• webapp/src/views/projects/translations/translationVisual/TranslationWithPlaceholders.tsx — inline highlights rendering + sortModifiers() for merging with glossary/ICU highlights
• webapp/src/views/projects/translations/translationVisual/GlossaryHighlight.tsx — pattern for inline underline + hover tooltip (QA highlights should coexist with glossary highlights)

Note: No CodeMirror editor decorations needed — live preview in the panel is sufficient during editing.

QA issue resolution

User actions on QA issues.

• IGNORE and UNIGNORE: Dismiss individual QA issues (persisted state, won't reappear on next check run)
• CORRECT: Frontend-only autofill — replaces the text span in the editor with the replacement value from the QA issue. User must still save manually. Only available when replacement is non-null.
• API endpoint for ignore/unignore action
• ~ API endpoint for correct action (for scripting purposes and similar; won't be used by frontend; same result as clicking "CORRECT" button on frontend and then saving the translation)
• I think I know how to do it
• I have validated my approach

~ Text comparison checks

Checks comparing translation against the base language text. Each check is a separate class. Each produces variant-specific messages with suggested corrections.

• Spaces mismatch (+extra spaces): 7 message variants — add/remove leading, trailing, doubled, non-breaking spaces. CJK-English spacing deferred (complex edge cases).
• Unmatched newlines: Mismatched empty lines (source vs. translation)
• Character case mismatch: 2 variants — capitalize or lowercase first letter
• Missing numbers: Numbers present in source but not in translation
• Punctuation mismatch: 3 variants — add, remove, or replace trailing punctuation (. , ! ? : ;)
• Brackets mismatch: 2 variants — different brackets, unbalanced brackets
• Special character mismatch: new paragraphs, currency signs, other characters
• Different URLs: 3 variants — add missing, remove extra, replace URL
• Inconsistent placeholders: different count, different order, or different number compared to source
• Inconsistent HTML: 2 variants — make consistent with source, fix HTML syntax
• I think I know how to do it
• I have validated my approach

~ Syntax checks

Checks on the translation text itself (no base comparison needed).

• ICU syntax validation (leverage existing formats/ parsing code)
• Repeated words detection ("When two identical words are repeated – even in the source")
• Empty translation detection (first slice, default: Off)
• Unresolved comments
• I think I know how to do it
• I have validated my approach

~ Spelling check

Spelling validation using an external service.

• Integration with LanguageTool or similar service
• ~ Don't allow enabling for unsupported languages in language specific QA settings
• I think I know how to do it
• I have validated my approach

QA filtering

Add QA-based filtering to the translation view.

• New "QA checks" submenu in the existing filter dropdown
• "Has QA issues" filter option
• All check types as individual checkboxes
• Visual separator between comparison checks (first 11; includes spelling check) and syntax checks (last 4)
• Integrate with existing filter menu pattern
• Backend query support for filtering by QA issues (join on QA issues table)
• I think I know how to do it
• I have validated my approach

Key files:

• webapp/src/views/projects/translations/TranslationFilters/ — existing filter system
• Backend translation query service — add QA filter parameters

QA settings

Per-project and per-language QA configuration.

• New "QA CHECKS" tab in Project Settings (alongside GENERAL and ADVANCED)
• Rule-based QA section: each check type has a Warning/Off dropdown (enum, not boolean)
• ~ AI QA checks section (below rule-based checks): list of user-defined AI checks with add/edit buttons and Off/Warning toggles (see "~ AI QA check" scope for details)
• Per-check defaults from Figma (5 default to Off, 10 to Warning — see table below)
• ~ QA for languages section: each language shows "Global QA" by default with gear icon to customize
  • ~ Per-language overrides: individually set Warning/Off per check
  • ~ Summary showing "Warnings: X, Off: Y" per language
  • ~ Option to turn all QA checks off for a language
• New backend entities: ProjectQaConfig (global) + LanguageQaConfig (per-language override)
• Info tooltips (ⓘ) on check types that need explanation
• I think I know how to do it
• I have validated my approach

Default values per check:

Check	Default
Empty translation	Off
Spaces mismatch	Warning
Unmatches new lines	Warning
Character case mismatch	Warning
Missing numbers	Warning
Spelling	Warning
Repeated words	Warning
Punctuation mismatch	Warning
Brackets mismatch	Warning
Special character mismatch	Off
Different URLs	Warning
Inconsistent placeholders	Warning
Inconsistent HTML	Warning
ICU syntax	Warning
Unresolved comments	Warning

Default values only apply to new projects. Existing projects need to have all QA checks off by default.

Batch QA recheck

Running QA checks across all existing translations.

• Batch job following existing batch pattern (same processor as on-save, but for bulk)
• Triggered when QA settings change (checks enabled/disabled)
• Uses existing BatchJobService.startJob() infrastructure
• When organization gains access to QA_CHECK feature, batch recheck all stale translations for all projects under the organization
• I think I know how to do it
• I have validated my approach

QA on dashboard

QA visibility on the project dashboard.

• QA badge with count on language rows (pink QA icon with issue count number; same component as QA badge on translations)
• Stale flag handling for aggregated views (icon represents multiple translations for a language):
  • If any translations have QA issues (regardless of stale) → normal icon, clickable to show list of all known QA issues
    • If any translation has stale QA checks → show loading spinner instead of number in badge
    • If no translation has stale QA checks → show badge with number representing how many qa issues are there
  • If no QA issues exist but any translations have qaChecksStale = true → gray icon with loading spinner instead of number in badge
  • If no QA issues and no stale translations → icon hidden
• ~ Popover on badge click showing:
  • ~ Issue breakdown by check type with counts (e.g., "Spelling: 2, Punctuation mismatch: 1")
  • ~ "SHOW" link per issue type → navigates to translation view filtered by that type + language
  • ~ "SHOW ALL" button → navigates to translation view showing all QA issues for that language
  • ~ "SETTINGS" link → navigates to QA settings page
• I think I know how to do it
• I have validated my approach

Key files: webapp/src/views/projects/dashboard/LanguageStats/

~ AI QA check

Important, but cuttable if it becomes a rabbit hole. Basic rule-based checks are higher priority.

Settings UI: Separate section in QA settings page, below the list of rule-based checks:

• List of user-defined AI QA checks (each with name/description)
• Add button to create a new AI check
• Edit button next to each existing AI check
• Off/Warning toggle per AI check (same pattern as rule-based checks)
• Add/Edit → navigates to AI playground panel for prompt creation and testing

AI Playground integration: Reuse the existing AI playground infrastructure, making it more generic:

• Make AI playground code support both translation mode (existing) and QA check mode (new)
• User can set custom prompt, enable/disable parts of system prompt
• Advanced mode: user writes completely custom prompt
• Result structure differs (QA issues instead of updated translations), but shared code for prompt editing, LLM provider selection, testing UI
• New backend entities for AI QA check configs (prompt, enabled state, etc.)

Runtime:

• Runs debounced in live preview mode (LLM calls are slow/expensive)
• Reuse existing AI playground / LLM provider infrastructure (AiPlaygroundResultService, AI_PLAYGROUND_TRANSLATE batch type)
• I think I know how to do it
• I have validated my approach

Key reference: StartBatchJobController — AI_PLAYGROUND_TRANSLATE pattern, AiPlaygroundResultService

Docs

• User documentation for QA checks feature
• Configuration guide for enabling/disabling checks
• I think I know how to do it
• I have validated my approach

TODOs & polish

Code quality, bug fixes, and missing pieces discovered during implementation.

• 1: QA stats performance — QA issue counts are fetched per-language for the dashboard. Consider adding counts to the LanguageStats model for single-query performance.
• 2: Default QA config for existing vs new projects — Existing projects should have QA checks disabled; new projects should use defaults. Must also be extensible for future QaCheckType additions.
• 3: Useless index on state — The Index(columnList = "state") on TranslationQaIssue is likely not useful (low cardinality). Consider removing.
• 4: Move projection interfaces — LanguageQaIssueCount and QaIssueCountByCheckType are inlined in the repository file. Move to own files.
• 5: Feature-gate QA issue count in query builder — addQaIssueCount() is called unconditionally. Should only run when QA_CHECKS feature is enabled.
• 6: Feature-gate QA issues in data provider — QA issues loaded regardless of feature flag. Add feature check before qaIssueRepository.findOpenByTranslationIds().
• 7: Extract QaCheckCategoryModel — Inline data class in controller. Move to separate file with proper HATEOAS assembler.
• 8: Wrap settings map in model — getSettings() returns raw Map. Wrap in proper DTO/model class. Same for other added endpoints.
• 9: Handle virtual issues on unignore — Virtual issues (from preview UI for not-yet-saved text) should be deleted rather than set back to OPEN on unignore.
• 10: Reusable QA test data — Create a dedicated QaTestData builder with pre-configured keys, translations, and QA scenarios reusable across all QA test classes.
• 11: Remove type cast for settings — as Record<QaCheckType, QaCheckSeverity> cast. Resolved once backend wraps settings map (see 8) and schema is regenerated.
• 12: Extract IssueRow component — IssueRow defined inline in QaBadgePopover.tsx. Move to separate file.
• 13: Use StyledComponent — Inline sx styles on "show" link Typography. Replace with Styled* component.
• 14: Extract IssueRow subcomponents — Consolidate duplicate/related rendering logic into IssueRow component.
• 15: Plural handling in QA preview — QA preview receives only single plural variant. Either send full ICU form or skip QA for plurals.
• 16: Feature-disabled empty state — "No issues" empty state should differ when QA feature is not enabled vs zero issues.
• 17: Plural limitation note — Add visible note in QA panel when editing plural form that some checks don't support plurals.
• 18: Fix generated schema type — Generated API schema incorrectly marks settings as non-nullable. Fix backend DTO type.
• 19: Extract CheckTypeFilterItem — Move to separate file from SubfilterQaChecks.tsx.
• 20: Extract CheckTypeFilterName — Move to separate file from SubfilterQaChecks.tsx.
• 21: Feature-gate QA button — QA issues button on translation cells shown unconditionally. Add feature flag check.
• 22: Implement handleCorrect — "Correct" action on highlighted QA issues not implemented. Needs to apply suggested fix and save via API.
• 23: QA badge doesn't update after batch job — QA badge/highlights don't reactively update when QA batch job finishes. Need refresh mechanism.
• 24: Plural-aware QA checks — Add isPlural flag to QaCheckParams. Checks that don't work with plurals should skip. Frontend should send full ICU message.
• 25: Reusable QA test data — (overlaps with 10) Reusable test data builder across all QA tests.
• 26: sortModifiers — short text edge case — Check what happens when translation text is shorter than a modifier's range.
• 27: Help text translations missing — QA check help text in project QA settings missing i18n translations.
• 28: Regenerate DB migrations — Consolidate all QA database migrations into clean changesets.
• 29: QaCheckItem — correct button styling — "Correct" button should be a full button with visible border.
• 30: QaCheckItem — button tooltips — Add tooltips to small "correct" and "ignore" icon buttons.
• 31: Demo project stale QA flags — Demo project creation leaves all translations with QA marked as stale.
• 32: Audit stale flag triggers — Ensure every code path that marks QA as stale also enqueues a recheck.
• 33: Feature flag gating audit — Audit QA feature flag is checked everywhere (settings page banner, API endpoints, etc.).
• 34: Bulk "rerun QA checks" operation — Add "Rerun QA checks" to bulk operations menu for translations.
• 35: E2E tests — Cypress E2E tests for QA feature (settings, badges, highlighting, actions, filters, bulk ops).
• 36: Review endpoint structure — Review QA API endpoints for logical REST resource hierarchy.
• 37: Partial QA checks rerun — When changing single QA setting, issues from other checks disappear. Likely logical issue.

Chowder

• PostHog analytics events for QA actions (check triggered, issue ignored, issue fixed)
• ~ Performance benchmarking for large projects

---

Sequence

First slice is complete. Remaining plan:

1. Week 2: Persisted mode (async, on save) + stale flag implementation + QA settings (global config) + implement a few quick text comparison checks (spaces, punctuation, case)
2. Week 3: More check implementations + QA in translation view (cell indicators + stale flag UX) + QA issue resolution + QA filtering + QA inline highlighting (read-only text)
3. Week 4: Batch QA recheck + QA on dashboard (with stale flag aggregation) + ~ AI QA check (with AI playground reuse) + ~ Per-language settings + Docs + scope hammering — cut nice-to-haves if needed, ship