BookInsight: "auto" language follows the book's own language#73
Closed
vitofico wants to merge 2 commits into
Closed
BookInsight: "auto" language follows the book's own language#73vitofico wants to merge 2 commits into
vitofico wants to merge 2 commits into
Conversation
The AI insight `language` style already defaulted to `auto`, but `auto` emitted no language clause at all — the model just picked (effectively English) even for a non-English book. Make `auto` write the insight in the book's own language. Server-side, in the prompt builder: - `auto` now resolves to the book's metadata language (`bundle.language`), normalized to ISO 639-1 (folds `en-US`/`eng` → `en`, validates two-letter codes against the canonical set). Explicit user codes still win; `auto` with no usable book language falls back to the old language-neutral prompt. - The resolved language steers ONLY the prompt body. The `book_insights` cache row is still keyed on the style value (`auto`), matching what the Android client caches under — so client and server never desync. - Bump PROMPT_VERSION 5 → 6 to regenerate stale `auto` rows. For scanned books (ISBN, no embedded language) the server backfills the language from OpenLibrary's edition-level Books API (`jscmd=details`, whose `details.languages` is edition-specific — unlike `/search.json`'s work-level `language`, which aggregates every translation a work ever had). The lookup is cached, best-effort, and never fails generation; it fires only when the bundle has an ISBN but no language and OpenLibrary is enabled, and never overrides a client-provided language. Library EPUBs already carry `<dc:language>`, so this is a no-op for them and no Android change is needed.
- retrieval: don't cache transient OpenLibrary failures (429/5xx) as a null language — only cache confirmed 200 responses, so a temporary outage no longer suppresses backfill for the whole TTL. - prompts: validate two-letter book-language tags against ISO_639_1_CODES so bogus metadata (e.g. "zz") can't become a prompt instruction. - service: pass `style or AiStyle()` when composing so a None style (which the cache key already treats as "auto") also gets auto-language resolution. - tests: assert the resolved book language never leaks into the cache key (row stays "auto"); cover transient-failure non-caching and bogus 2-letter rejection; give the shared FakeRetriever the new protocol method. - update stale "auto emits no clause" comments; bump the prompt-version regression pin 5 -> 6; apply ruff format.
Owner
Author
|
Superseded by #74 — this PR auto-closed when the branch was renamed from |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The AI-insight
languagestyle already defaulted toauto, butautoemitted no language clause — the model just picked (effectively English) even for a non-English book. This makesautowrite the insight in the book's own language.How
Prompt resolution (
core/ai/prompts.py)autoresolves to the book's metadata language (bundle.language), normalized to ISO 639-1 — foldsen-US/eng→en, and validates two-letter codes against the canonicalISO_639_1_CODESset so bogus tags never reach the model.autowith no usable book language falls back to the prior language-neutral prompt.book_insightsrow is still keyed on the style value (auto) — matching what the Android client caches under — so client and server never desync. (Regression-tested.)PROMPT_VERSIONbumped 5 → 6 to regenerate staleautorows (they were generated with no language clause).Scanned-book backfill (
core/ai/retrieval.py,core/ai/service.py)lookup_book_language(isbn)uses OpenLibrary's edition-level Books API (jscmd=details). Itsdetails.languagesis edition-specific — unlike/search.json'slanguage, which is work-level and aggregates every translation a work ever had (verified against the live API), so it can't identify the scanned edition.bundle.languageonly when the bundle has an ISBN but no language and OpenLibrary is enabled, on cache-miss only. It's cached (positive + confirmed-negative, but not transient 429/5xx failures), best-effort, and never fails generation. It never overrides a client-provided language.<dc:language>, so this is a no-op for them and no Android change is needed — the in-flight ISBN-scan feature (Book scan: camera/ISBN capture → reader-affinity → optional insight (Android) #71) benefits for free since it already sends the ISBN.Operational note
Bumping
PROMPT_VERSIONinvalidates all cached insights (including explicit-language ones that are technically unaffected), so they regenerate on next open — a one-time generation-quota cost. It's the only cache-bust lever available.No DB migration: the
languagecolumn already exists; only its prompt semantics changed.Tests
348unit tests pass (new coverage for auto-resolution, ISO normalization/validation, the edition-level lookup, transient-failure non-caching, and the cache-key invariant) plus the AI integration suite.ruffclean.