Merge pull request #4 from bluedynamics/feature/fulltext-search-v2

Language-aware full-text search + Title/Description fix

Author: jensens

ARCHITECTURE.md

@@ -41,6 +41,7 @@ Indexes are discovered from ZCatalog at startup, not hardcoded.
 4. `IndexRegistry.sync_from_catalog()` reads `catalog._catalog.indexes`
 5. Each index's `meta_type` is mapped via `META_TYPE_MAP` to an `IndexType` enum
 6. `_register_dri_translators()` discovers `DateRecurringIndex` instances and registers `IPGIndexTranslator` utilities
+7. `_ensure_text_indexes()` creates GIN expression indexes for any dynamically discovered `TEXT`-type indexes with `idx_key is not None` (Title, Description, addon ZCTextIndex fields)
 
 ### IndexType Enum
 
@@ -215,7 +216,9 @@ _HANDLERS = {
 
 **PathIndex**: `idx->>'path' LIKE '/plone/folder/%'` (subtree), `idx->>'path_parent' = '/plone/folder'` (children), navtree breadcrumb queries.
 
-**TextIndex**: `searchable_text @@ plainto_tsquery('simple', %(text)s)` for SearchableText; other text indexes treated as field match.
+**TextIndex (SearchableText)**: `searchable_text @@ plainto_tsquery(pgcatalog_lang_to_regconfig(%(lang)s)::regconfig, %(text)s)` -- language-aware stemming via the per-object `Language` field. Falls back to `'simple'` when no language is set.
+
+**TextIndex (Title/Description/addon)**: `to_tsvector('simple'::regconfig, COALESCE(idx->>'Title', '')) @@ plainto_tsquery('simple'::regconfig, %(text)s)` -- word-level matching on idx JSONB values, backed by GIN expression indexes. Uses `'simple'` config (no stemming) because expression indexes require a fixed regconfig.
 
 ## Transactional Writes
 
@@ -239,12 +242,46 @@ Includes:
 
 - `ALTER TABLE object_state ADD COLUMN IF NOT EXISTS ...` for catalog columns (`path`, `idx`, `searchable_text`)
 - `pgcatalog_to_timestamptz()` immutable wrapper for expression indexes
+- `pgcatalog_lang_to_regconfig()` maps Plone language codes (ISO 639-1) to PG text search configurations (e.g. `'de'` → `'german'`). Used at both write time (`to_tsvector`) and query time (`plainto_tsquery`). Returns `'simple'` for NULL, empty, or unmapped languages.
 - GIN index on `idx` JSONB
 - B-tree expression indexes on `idx` JSONB for path queries (`path`, `path_parent`, `path_depth`)
 - B-tree expression indexes for common sort/filter fields (modified, created, effective, expires, sortable_title, portal_type, review_state, UID)
 - Full-text GIN index on `searchable_text`
+- GIN expression indexes for Title/Description tsvector matching (`to_tsvector('simple', COALESCE(idx->>'Title', ''))`)
+- Dynamic GIN expression indexes for addon ZCTextIndex fields (created at startup by `_ensure_text_indexes()`)
 - rrule_plpgsql schema and functions (for DateRecurringIndex)
 
+## Full-Text Search
+
+Three tiers of text search, each with different characteristics:
+
+### SearchableText (Language-Aware)
+
+Uses the dedicated `searchable_text` TSVECTOR column with per-object language stemming:
+
+- **Write path**: `to_tsvector(pgcatalog_lang_to_regconfig(idx->>'Language')::regconfig, text)` -- language extracted from the object's `Language` field in idx JSONB
+- **Query path**: `searchable_text @@ plainto_tsquery(pgcatalog_lang_to_regconfig(%(lang)s)::regconfig, %(text)s)` -- language from the query's `Language` filter
+- **Index**: GIN on `searchable_text` column
+- **Stemming**: Yes, for the 30 supported languages (falls back to `'simple'` for unknown/empty)
+
+### Title / Description (Word-Level)
+
+Uses tsvector expression matching on idx JSONB values:
+
+- **Write path**: Values stored as plain text in `idx->>'Title'` / `idx->>'Description'`
+- **Query path**: `to_tsvector('simple', COALESCE(idx->>'Title', '')) @@ plainto_tsquery('simple', %(text)s)`
+- **Index**: GIN expression indexes (pre-created in DDL)
+- **Stemming**: No (`'simple'` config) -- expression indexes require a fixed regconfig. Language-aware stemmed search for titles is available via SearchableText (which includes title text).
+
+### Addon ZCTextIndex Fields
+
+Any addon that registers a ZCTextIndex in ZCatalog (via `catalog.xml`) is automatically supported:
+
+1. `sync_from_catalog()` discovers the index → registered as `(IndexType.TEXT, idx_key, source_attrs)`
+2. `_ensure_text_indexes()` creates a GIN expression index at startup: `to_tsvector('simple', COALESCE(idx->>'{idx_key}', ''))`
+3. Value extracted into idx JSONB during indexing (idx_key is not None)
+4. `_handle_text()` generates tsvector expression matching -- zero addon code needed
+
 ## Query Optimizations
 
 1. **orjson**: Registered as psycopg's JSONB deserializer for faster JSON parsing

CHANGES.md

@@ -1,5 +1,34 @@
 # Changelog
 
+## 1.0.0b4
+
+### Added
+
+- **Language-aware full-text search**: SearchableText now uses per-object
+  language for stemming. The `pgcatalog_lang_to_regconfig()` PL/pgSQL function
+  maps Plone language codes (ISO 639-1, 30 languages) to PostgreSQL text search
+  configurations (e.g. `"de"` → `german`). Falls back to `'simple'` for
+  unmapped or missing languages. Non-multilingual sites are unaffected.
+
+  Python mirror: `columns.language_to_regconfig()` for testing/validation.
+
+- **Title/Description text search**: Title and Description queries now use
+  tsvector word-level matching instead of exact JSONB containment.
+  `catalog(Title="Hello")` now correctly matches `"Hello World"`.
+  Backed by GIN expression indexes with `'simple'` config (no stemming).
+
+- **Automatic addon ZCTextIndex support**: Addon-registered ZCTextIndex fields
+  are automatically discovered at startup. GIN expression indexes are created
+  dynamically by `_ensure_text_indexes()`, and queries use tsvector matching --
+  zero addon code needed.
+
+### Fixed
+
+- **Title/Description query broken**: Previously, querying Title or Description
+  as ZCTextIndex used JSONB exact containment (`idx @> '{"Title":"Hello"}'`),
+  which only matched exact values, not words within text. Now uses
+  `to_tsvector`/`plainto_tsquery` for proper word-level matching.
+
 ## 1.0.0b3
 
 ### Fixed

README.md

@@ -11,7 +11,7 @@ Requires [zodb-pgjsonb](https://github.com/bluedynamics/zodb-pgjsonb) as the ZOD
 - **Extensible** via `IPGIndexTranslator` named utilities for custom index types
 - **Dynamic index discovery** from ZCatalog at startup -- addons adding indexes via `catalog.xml` just work
 - **Transactional writes** -- catalog data written atomically alongside object state during ZODB commit
-- **Full-text search** via PostgreSQL `tsvector`/`tsquery`
+- **Full-text search** via PostgreSQL `tsvector`/`tsquery` -- language-aware stemming for SearchableText (30 languages), word-level matching for Title/Description/addon ZCTextIndex fields
 - **Zero ZODB cache pressure** -- no BTree/Bucket objects stored in ZODB
 - **Container-friendly** -- works on standard `postgres:17` Docker images, no extensions required
 
@@ -51,6 +51,8 @@ Once installed, `portal_catalog` is replaced with `PlonePGCatalogTool`. All cata
 results = catalog(portal_type="Document", review_state="published")
 results = catalog(Subject={"query": ["Python", "Plone"], "operator": "or"})
 results = catalog(SearchableText="my search term")
+results = catalog(SearchableText="Katzen", Language="de")  # language-aware stemming
+results = catalog(Title="quick fox")  # word-level match (finds "The Quick Brown Fox")
 results = catalog(path={"query": "/plone/folder", "depth": 1})
 
 # Recurring events (DateRecurringIndex)

src/plone/pgcatalog/columns.py

@@ -269,6 +269,61 @@ def _convert_zope_datetime(dt):
     return dt.ISO8601()
 
 
+# --------------------------------------------------------------------------
+# Language → PG text search configuration mapping
+# --------------------------------------------------------------------------
+
+# Plone language code (ISO 639-1) → PostgreSQL regconfig name.
+# Mirrors the SQL function pgcatalog_lang_to_regconfig() in schema.py.
+_LANG_TO_REGCONFIG = {
+    "ar": "arabic",
+    "hy": "armenian",
+    "eu": "basque",
+    "ca": "catalan",
+    "da": "danish",
+    "nl": "dutch",
+    "en": "english",
+    "fi": "finnish",
+    "fr": "french",
+    "de": "german",
+    "el": "greek",
+    "hi": "hindi",
+    "hu": "hungarian",
+    "id": "indonesian",
+    "ga": "irish",
+    "it": "italian",
+    "lt": "lithuanian",
+    "ne": "nepali",
+    "nb": "norwegian",
+    "nn": "norwegian",
+    "no": "norwegian",
+    "pt": "portuguese",
+    "ro": "romanian",
+    "ru": "russian",
+    "sr": "serbian",
+    "es": "spanish",
+    "sv": "swedish",
+    "ta": "tamil",
+    "tr": "turkish",
+    "yi": "yiddish",
+}
+
+
+def language_to_regconfig(lang):
+    """Map a Plone language code to a PG text search configuration name.
+
+    Args:
+        lang: Plone language code (e.g. "de", "en-us") or None/""
+
+    Returns:
+        PG regconfig name (e.g. "german", "english") or "simple"
+    """
+    if not lang:
+        return "simple"
+    base = lang.lower().split("-")[0].split("_")[0]
+    return _LANG_TO_REGCONFIG.get(base, "simple")
+
+
 # --------------------------------------------------------------------------
 # Path utilities
 # --------------------------------------------------------------------------

src/plone/pgcatalog/config.py

@@ -310,7 +310,9 @@ def get_extra_columns(self):
             ExtraColumn("idx", "%(idx)s"),
             ExtraColumn(
                 "searchable_text",
-                "to_tsvector('simple'::regconfig, %(searchable_text)s)",
+                "to_tsvector("
+                "pgcatalog_lang_to_regconfig(%(idx)s::jsonb->>'Language')"
+                "::regconfig, %(searchable_text)s)",
             ),
         ]
 
@@ -324,9 +326,16 @@ def get_schema_sql(self):
         from plone.pgcatalog.schema import CATALOG_COLUMNS
         from plone.pgcatalog.schema import CATALOG_FUNCTIONS
         from plone.pgcatalog.schema import CATALOG_INDEXES
+        from plone.pgcatalog.schema import CATALOG_LANG_FUNCTION
         from plone.pgcatalog.schema import RRULE_FUNCTIONS
 
-        return CATALOG_COLUMNS + CATALOG_FUNCTIONS + CATALOG_INDEXES + RRULE_FUNCTIONS
+        return (
+            CATALOG_COLUMNS
+            + CATALOG_FUNCTIONS
+            + CATALOG_LANG_FUNCTION
+            + CATALOG_INDEXES
+            + RRULE_FUNCTIONS
+        )
 
     def process(self, zoid, class_mod, class_name, state):
         # Look up pending data from the thread-local store (set by
@@ -393,10 +402,55 @@ def register_catalog_processor(event):
         storage.register_state_processor(processor)
         log.info("Registered CatalogStateProcessor on %s", storage)
         _sync_registry_from_db(db)
+        _ensure_text_indexes(storage)
     else:
         log.debug("Storage %s does not support state processors", storage)
 
 
+def _ensure_text_indexes(storage):
+    """Create GIN expression indexes for dynamically discovered TEXT indexes.
+
+    For each TEXT-type index with idx_key != None (not SearchableText),
+    creates a GIN expression index on to_tsvector('simple', idx->>'{key}')
+    if it doesn't already exist.  Uses an autocommit connection to avoid
+    REPEATABLE READ lock conflicts.
+    """
+    from plone.pgcatalog.columns import get_registry
+    from plone.pgcatalog.columns import IndexType
+    from plone.pgcatalog.columns import validate_identifier
+
+    registry = get_registry()
+    text_indexes = [
+        (name, idx_key)
+        for name, (idx_type, idx_key, _) in registry.items()
+        if idx_type == IndexType.TEXT and idx_key is not None
+    ]
+    if not text_indexes:
+        return
+
+    dsn = getattr(storage, "_dsn", None)
+    if not dsn:
+        return
+
+    import psycopg
+
+    try:
+        with psycopg.connect(dsn, autocommit=True) as conn:
+            for name, idx_key in text_indexes:
+                validate_identifier(idx_key)
+                idx_name = f"idx_os_cat_{idx_key.lower()}_tsv"
+                conn.execute(
+                    f"CREATE INDEX IF NOT EXISTS {idx_name} "
+                    f"ON object_state USING gin ("
+                    f"to_tsvector('simple'::regconfig, "
+                    f"COALESCE(idx->>'{idx_key}', ''))) "
+                    f"WHERE idx IS NOT NULL"
+                )
+                log.info("Ensured GIN text index %s for %s", idx_name, name)
+    except Exception:
+        log.warning("Failed to create text expression indexes", exc_info=True)
+
+
 def _register_dri_translators(catalog):
     """Discover DateRecurringIndex instances and register IPGIndexTranslator utilities.
 

src/plone/pgcatalog/query.py

@@ -171,6 +171,9 @@ def result(self):
         }
 
     def process(self, query_dict):
+        # Store full query dict for cross-index lookups (e.g. Language)
+        self._query = query_dict
+
         # Always filter for cataloged objects
         self.clauses.append("idx IS NOT NULL")
 
@@ -388,19 +391,34 @@ def _handle_text(self, name, idx_key, spec):
             return
 
         if idx_key is None:
-            # SearchableText → tsvector full-text search
+            # SearchableText → dedicated tsvector column, language-aware.
+            # Uses pgcatalog_lang_to_regconfig() SQL function to map
+            # Plone language codes to PG regconfig names at query time.
             p_text = self._pname("text")
             p_lang = self._pname("lang")
             self.clauses.append(
-                f"searchable_text @@ plainto_tsquery(%({p_lang})s::regconfig, %({p_text})s)"
+                f"searchable_text @@ plainto_tsquery("
+                f"pgcatalog_lang_to_regconfig(%({p_lang})s)::regconfig, "
+                f"%({p_text})s)"
             )
             self.params[p_text] = str(query_val)
-            self.params[p_lang] = "simple"
+            # Extract Language from the query dict if present
+            lang_val = self._query.get("Language")
+            if isinstance(lang_val, dict):
+                lang_val = lang_val.get("query", "")
+            self.params[p_lang] = str(lang_val) if lang_val else ""
         else:
-            # Title / Description — treat as field exact match
+            # Title / Description / addon ZCTextIndex →
+            # tsvector expression on idx JSONB, 'simple' config.
+            # Expression matches the GIN index created in schema.py /
+            # _ensure_text_indexes() for index-backed queries.
             p = self._pname(name)
-            self.clauses.append(f"idx @> %({p})s::jsonb")
-            self.params[p] = Json({idx_key: query_val})
+            self.clauses.append(
+                f"to_tsvector('simple'::regconfig, "
+                f"COALESCE(idx->>'{idx_key}', '')) "
+                f"@@ plainto_tsquery('simple'::regconfig, %({p})s)"
+            )
+            self.params[p] = str(query_val)
 
     # -- ExtendedPathIndex --------------------------------------------------