Keep the library working when its custom-column definitions can't be read (#1154)

get_cc_columns reads a library's custom-column definitions for five surfaces:
the book detail page, the detail API, the books table and both search screens.
Only the API had error handling, and it caught the wrong class -- an unreadable
schema raises SQLAlchemy's OperationalError, so a missing custom_columns table
took whole pages down with a 500 while the rest of the library read fine.

Degrade inside get_cc_columns instead of copying a guard to five callsites, and
read under no_autoflush so the lookup can never flush a caller's pending write
and swallow their error as "no custom columns".

Verified on a live server with custom_columns renamed out from under it:
/book/2 and /api/v1/books/2 went 500 -> 200, /table and /advsearch stayed 200,
and subsequent queries on the same session kept working. Suite 9F/4189P ->
3F/4199P, the 3 remaining pre-existing and unrelated.

Addresses #1153.

Author: new-usemame

CHANGELOG.md

@@ -21,6 +21,8 @@ is for things you can see or feel when running the app.
 - **Two people using the library at once no longer break each other's pages.** The server kept one working copy of your library data and shared it across every request it was handling. As soon as any one page finished, it closed that copy, so any other page still being built lost the books it had already read and failed. In the log it showed up as `Instance ... is not persistent within this Session`, pointing at whatever the unlucky page happened to be doing at the time, which made it look like a different bug on every occurrence. It got likelier the busier the server was, so it hit big libraries, shared instances and anything running during an import hardest. Each request now keeps its own working copy ([#1150](https://github.com/new-usemame/Calibre-Web-NextGen/issues/1150)).
 - **A page you are reading no longer breaks because a background job finished.** Duplicate scans, thumbnail generation, metadata backups and Hardcover syncs all share the server's connection to your library, and when one of them finished it could close that connection while a page you had open was still reading from it. The page then failed, usually with `Cannot operate on a closed database` in the log. It was intermittent by nature — it only bit when the timing overlapped, which is why a manual scan could break browsing once and then work fine on the next try. Background jobs and page loads now keep their own handle on the library, so one finishing can no longer interrupt the other ([#1121](https://github.com/new-usemame/Calibre-Web-NextGen/issues/1121), reported downstream of [#1048](https://github.com/new-usemame/Calibre-Web-NextGen/issues/1048) by [@auspex](https://github.com/auspex)).
 
+- **Your library keeps working when its extra fields can't be read.** Book pages, the books table and both search screens all ask the library for its custom-column definitions — the extra fields you can add to a book in Calibre. If that lookup failed because the library was mid-write, its folder had moved, or the definitions table wasn't there yet, those pages returned an error instead of simply leaving the extra fields out, even though the rest of the library was perfectly readable. Custom columns are supplementary, so these pages now load without them and note the reason in the log ([#1153](https://github.com/new-usemame/Calibre-Web-NextGen/issues/1153)).
+
 - **Opening your library no longer runs the same search twice.** Every visit to the library asked the server for the first page of books, then immediately asked for it again at a different size and threw the first answer away. Nothing looked wrong — the wasted answer never reached the screen — but on a large library that first page is the slowest query on it, and it was being run twice on every load, for every reader. The grid now waits until it knows how many columns it has before asking, so the page is fetched once ([#1144](https://github.com/new-usemame/Calibre-Web-NextGen/issues/1144)).
 
 ## [v4.1.21] - 2026-07-25

CHANGES-vs-upstream.md

@@ -407,6 +407,8 @@ Format: each row is one fork-PR, mapped to its upstream PR or issue (if any), wi
 | #340 | **A new cover now shows up everywhere immediately — web UI and Kobo.** Applying a cover via the focused cover-picker (`POST /book/<id>/cover/apply`) saved `has_cover` and busted the picker's own `?ts=` preview but never bumped `book.last_modified`. Every cover URL in the app cache-busts on `c=last_modified` (the `jinjia` filter feeding detail page, home-grid srcset, edit page, and og:image), so an unchanged timestamp meant the browser kept serving the cached cover until a manual refresh; Kobo native sync selects changed books with `Books.last_modified > sync_token.books_last_modified`, so the device never re-pulled the new cover. **Root cause confirmed on teenyverse**: *Carmilla*'s `cover.jpg` was rewritten 2026-05-28 17:06Z while its `last_modified` was stuck at 2026-05-27 03:01Z. **Fix** mirrors the edit-book path in `_apply_response` — bump `last_modified`, `set_metadata_dirty`, commit, then `remove_synced_book(all=True)`; the `CoverImageId` already embeds the cover mtime, so devices re-fetch once the book is re-selected. The post-commit best-effort steps were split from the critical DB write so a commit failure now returns 500 instead of a false `ok:True`. **Discoverability**: the cover image itself is now the entry to the picker on the book detail page — an always-visible corner camera badge (discoverable on touch) plus a hover/focus "Change cover" overlay; the edit-page button was relabeled from "Open focused cover picker" to "Change cover" (no overlay on the edit page, to avoid discarding unsaved form edits on an accidental click). Shared overlay CSS in `cwa.css` scoped to a new `.cover-editable` class; reuses the existing "Change cover" message id (no new translation strings). **4 regression tests** pin the three sync/cache invariants (last_modified bump, `remove_synced_book(all=True)`, `set_metadata_dirty`) for both the success and failure paths plus a source-pin refactor guard. **Live-verified on cwn-local**: real authenticated multipart upload → `last_modified` bumped, seeded `kobo_synced_books` row cleared, `metadata_dirtied` set, detail page served the new cover (new `c=` param, new image bytes), `CoverImageId` recomputed to `<uuid>-<mtime>`; both entry points exercised via Playwright with 0 console errors. Greptile reviewed (4/5); both P2 findings addressed. | `150e83582` | v4.0.141 |
 | #349 | **Edit highlights in the web reader + a KOReader↔server annotation bridge (Phase 1 + 2 of the Kobo highlight feature).** Phase 1: select text in the epub reader → popup (4 colour swatches + optional note) → Save creates a `source='webreader'` highlight that paints as an overlay and persists; click a highlight to recolor / edit the note / soft-delete. Routes on the `annotations` blueprint, UI in `annotations.js`, Hardcover fan-out when that sync target is enabled. Phase 2: `GET/PUT /kosync/syncs/annotations` on the `kosync` blueprint (reuses its Basic auth + checksum→book resolution, no new credentials), a `device_origin_id` column + idempotent migration to suppress feedback loops, a portable-annotation projection as the wire shape, and a `cwasync.koplugin` KoboReader.sqlite provider that writes highlights into a Kobo so the stock reader renders them. The plugin's device-write toggle ships **default-off** until the on-device round-trip (`notes/feat-annotation-koreader-bridge-device-verification.md` Tests 1–4) passes — the SQL→Nickel render was already proven on real hardware (2026-05-26); only the plugin's runtime sqlite-FFI/wifi glue is ungated. Phase 3 (server→stock-Nickel push) excluded — disproven on hardware. **i18n**: the popup strings move from hardcoded English to the reader's `window.calibre.annotationsI18n` bridge (each `_()`-wrapped); Save/Cancel/Delete + the empty-state reuse existing msgids, verified live under a Hungarian session (`Mentés`/`Mégsem`/`Törlés`). **Verified**: 38 unit + a provider schema-fit guard replaying the exact INSERT/SELECT against the real `Bookmark` schema (and a real 1,940-bookmark device-DB copy) + busted Lua + live HTTP CRUD and pull/push (503/401/200) + full Playwright create→paint→reload→edit→delete. Also de-stales two plugin tests already red on `main` (plugin version `4.0.54`→`4.0.136`; CWA→NextGen prompt rebrand). | `(pending squash)` | `(pending release)` |
 
+| #TBD | (fork issue [#1153](https://github.com/new-usemame/Calibre-Web-NextGen/issues/1153), self-reported) | **A book's page no longer 500s when the library's custom-column definitions can't be read.** `_detail_custom_columns()` in `cps/api/books.py` documents itself as "degrading safely if DB metadata is unavailable" but caught only `AttributeError`/`KeyError`/`TypeError`. The error an actually-unreadable calibre schema raises is a SQLAlchemy `OperationalError` (`no such table: custom_columns`) — not in that list — so it escaped and took the whole detail payload down over a supplementary field. Reachable whenever the metadata DB opens but its schema isn't there: library mid-write, moved/renamed library path, `custom_columns` mid-migration, partially-restored library. **Pre-existing in production**; it was masked in tests only because `calibre_db.session` could be `None` and `None.query(...)` raises `AttributeError`, so the *no-session* case was caught while the *unreadable-schema* case was never exercised. #1149 made `session` a property that always materialises, which removed that accidental cushion and turned the latent gap into two red tests on `main` (`test_587_koreader_progress`, plus the same class in `test_kobo_bookmark_created_at` / `test_api_v1_books` under `--dist=loadfile` worker grouping). **Fix**: degrade inside `CalibreDB.get_cc_columns` itself (catch `SQLAlchemyError`/`AttributeError` → log + return `[]`), not at one callsite. Driving the real routes on `cwn-local` showed the API-only fix was incomplete: `/book/<id>` (`cps/web.py` `show_book`) still 500'd, because four of the five callers — book detail page, books table, and both search surfaces — had no guard at all. All five want "the custom columns, if any" and none can act on a DB error, so one source of truth beats five copies of the same try/except. `_detail_custom_columns` keeps its own guard as defense-in-depth. **4 regression tests** in `tests/unit/test_book_detail_custom_columns_degrade.py` — the contract pin (unreadable schema degrades to `[]`) is red without the fix and green with it; two supporting pins assert the fixture reproduces the real raise and that `session` materialises rather than returning `None`, so the contract test can't pass vacuously. Fixture saves/restores the class-level `session_factory` — the bug class here is cross-test pollution. | `TBD` | `TBD` |
+
 ## Backports from janeczku/calibre-web (2026-04 wave)
 
 CWA detached from the original Calibre-Web upstream (`janeczku/calibre-web`) at some point before this fork started. The original upstream is alive and pushing fixes — most notably a security wave in April 2026 by @jvoisin and @haraldpdl that CWA never picked up. We sync these fixes directly into our `main` so users on `ghcr.io/new-usemame/calibre-web-nextgen` get them without waiting for upstream's next release tag (releases on calibre-web tend to be 5–6 months apart).

cps/api/books.py

@@ -6,6 +6,7 @@
 from flask import jsonify, request
 from flask_babel import get_locale
 from sqlalchemy import and_, func, or_
+from sqlalchemy.exc import SQLAlchemyError
 from sqlalchemy.sql.functions import coalesce
 
 from . import api_v1
@@ -39,10 +40,21 @@ def _visible_hot_book_ids(book_ids):
 
 
 def _detail_custom_columns():
-    """Classic-parity display definitions, degrading safely if DB metadata is unavailable."""
+    """Classic-parity display definitions, degrading safely if DB metadata is unavailable.
+
+    ``SQLAlchemyError`` is the case that actually happens in production: the
+    calibre metadata DB is reachable but its schema is not (a library that is
+    still being written, a wrong/renamed library path, a mid-migration
+    ``custom_columns``), and the query raises ``OperationalError``. The
+    non-DB errors below cover the reconnect window, where ``calibre_db.session``
+    can be absent rather than merely unreadable.
+
+    Custom columns are supplementary to a book's detail payload, so an
+    unreadable definition table must not take the whole page down with a 500.
+    """
     try:
         return calibre_db.get_cc_columns(config, filter_config_custom_read=True)
-    except (AttributeError, KeyError, TypeError):
+    except (SQLAlchemyError, AttributeError, KeyError, TypeError):
         log.warning("Custom-column definitions unavailable for book detail", exc_info=True)
         return []
 

cps/db.py

@@ -23,7 +23,7 @@
 from sqlalchemy.orm import relationship, sessionmaker, scoped_session, joinedload, object_session
 from sqlalchemy.orm.collections import InstrumentedList
 from sqlalchemy.ext.declarative import DeclarativeMeta
-from sqlalchemy.exc import OperationalError
+from sqlalchemy.exc import OperationalError, SQLAlchemyError
 try:
     # Compatibility with sqlalchemy 2.0
     from sqlalchemy.orm import declarative_base
@@ -1887,8 +1887,43 @@ def search_query(self, term, config, *join, allow_show_hidden=False):
             .filter(or_(*filter_expression))
 
     def get_cc_columns(self, config, filter_config_custom_read=False):
-        self.ensure_session()
-        tmp_cc = self.session.query(CustomColumns).filter(CustomColumns.datatype.notin_(cc_exceptions)).all()
+        """Custom-column display definitions, or ``[]`` if they can't be read.
+
+        Degrading here rather than at each callsite is deliberate: five callers
+        (book detail page, book detail API, books table, and both search
+        surfaces) all want "the custom columns, if there are any", and none of
+        them can do anything useful with a DB error. Letting it propagate meant
+        an unreadable ``custom_columns`` table took whole pages down with a 500
+        over supplementary fields -- while the rest of the library read fine.
+
+        That happens whenever the metadata DB opens but its schema does not:
+        a library mid-write, a moved or renamed library path, ``custom_columns``
+        mid-migration, a partly restored library. An empty list is the same
+        thing these callers already handle for a library with no custom columns
+        at all, so the degraded page is a shape they all render correctly.
+        """
+        try:
+            self.ensure_session()
+            session = self.session
+            # no_autoflush because this is a read of definitions, and the
+            # session may be carrying a caller's pending mutation. With
+            # autoflush on, our SELECT would flush *their* write first, so an
+            # unrelated IntegrityError would surface here, get swallowed as
+            # "no custom columns", and leave the session needing a rollback for
+            # the rest of the request. Not reachable from today's five callers
+            # (all read paths), but the failure would be silent and remote from
+            # its cause, so don't leave it available to the sixth.
+            with session.no_autoflush:
+                tmp_cc = (session.query(CustomColumns)
+                          .filter(CustomColumns.datatype.notin_(cc_exceptions)).all())
+        except (SQLAlchemyError, AttributeError):
+            # AttributeError: the session is absent rather than unreadable --
+            # `session` is None whenever session_factory is (before init_db, or
+            # after an explicit `session = None`), and None.query() is an
+            # AttributeError rather than a SQLAlchemyError.
+            log.warning("Custom-column definitions unavailable; continuing without them",
+                        exc_info=True)
+            return []
         cc = []
         r = None
         if config.config_columns_to_ignore: