Update docs to reflect @meta metadata type preservation (#23)

Document the two-tier idx structure, @meta codec encoding, and
brain attribute resolution order across architecture, query-api,
schema, and zcatalog-compat reference pages.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author: jensens

CHANGES.md

@@ -1,5 +1,18 @@
 # Changelog
 
+## 1.0.0b13
+
+### Fixed
+
+- Preserve original Python types for metadata columns (e.g. `brain.effective`
+  now returns a Zope `DateTime` object instead of an ISO string).
+  Non-JSON-native metadata values (DateTime, datetime, date, etc.) are
+  encoded via the Rust codec into `idx["@meta"]` at write time and restored
+  on brain attribute access with per-brain caching.  JSON-native values
+  (str, int, float, bool, None) remain in top-level `idx` unchanged.
+  Backward compatible — old data without `@meta` still works.
+  Fixes #23.
+
 ## 1.0.0b12
 
 ### Fixed
@@ -12,7 +25,7 @@
   the portal root before traversal (matching Plone's `CatalogTool`).
   Fixes #21.
 
-## 1.0.0b11 
+## 1.0.0b11
 
 ### Fixed
 
@@ -23,12 +36,12 @@
 - Fix ZMI "Update Catalog" and "Clear and Rebuild" buttons returning 404.
   Added missing `manage_catalogReindex` and `manage_catalogRebuild` methods.
   Fixes #19.
-  
+
 - Fix `clearFindAndRebuild` indexing non-content objects (e.g. `acl_users`).
   Now filters for contentish objects only (those with a `reindexObject` method),
   matching Plone's `CatalogTool` behavior.
   Fixes #20.
-  
+
 ### Changed
 
 - `uniqueValuesFor(name)` is now a supported API (no longer deprecated).

docs/sources/explanation/architecture.md

@@ -205,7 +205,10 @@ query, and returns lightweight `_PendingBrain` instances with just enough interf
 6. **Results are wrapped in `CatalogSearchResults` with `PGCatalogBrain` objects.**
    Each brain is a lightweight wrapper around a PG row dict. It implements
    `ICatalogBrain` for Plone compatibility and supports attribute access into the
-   `idx` JSONB for catalog metadata.
+   `idx` JSONB for catalog metadata. Non-JSON-native metadata (such as Zope
+   `DateTime` objects) is stored under `idx["@meta"]` via the Rust codec and
+   decoded on first access with per-brain caching (see {doc}`../reference/schema`
+   for the `@meta` structure).
 
 ## Lazy loading
 
@@ -309,9 +312,11 @@ with PG-backed implementations:
   mapping table.
 
 - **Brain attribute resolution** distinguishes known from unknown fields. Known
-  catalog fields (registered indexes or metadata) return `None` when absent from the
-  `idx` JSONB -- matching ZCatalog's Missing Value behavior. Unknown fields raise
-  `AttributeError`, which triggers the `getObject()` fallback in
+  catalog fields (registered indexes or metadata) are resolved first from the
+  `idx["@meta"]` dict (for non-JSON-native types like `DateTime`), then from the
+  top-level `idx` JSONB.  Fields missing from both return `None` -- matching
+  ZCatalog's Missing Value behavior.  Unknown fields raise `AttributeError`,
+  which triggers the `getObject()` fallback in
   `CatalogContentListingObject.__getattr__()`.
 
 - **Blocked methods**: ZCatalog methods that would return wrong/empty data

docs/sources/reference/query-api.md

@@ -338,9 +338,16 @@ Lightweight result object backed by a PostgreSQL row. Implements
 All registered indexes and metadata columns are accessible as
 attributes (e.g., `brain.portal_type`, `brain.Title`, `brain.Subject`).
 
+- Non-JSON-native metadata (Zope `DateTime`, `datetime`, `date`, etc.)
+  is stored in `idx["@meta"]` via the Rust codec and decoded on first
+  access with per-brain caching. This means `brain.effective` returns
+  a `DateTime` object, not an ISO string. See {doc}`schema` for the
+  `@meta` JSONB structure.
+- JSON-native metadata (str, int, float, bool, None, lists/dicts of
+  these) is stored directly in the top-level `idx` JSONB.
 - For registered indexes/metadata: returns `None` if the field is
-  missing from the `idx` JSONB (Missing Value behavior, matching
-  ZCatalog).
+  missing from both `@meta` and top-level `idx` (Missing Value
+  behavior, matching ZCatalog).
 - For unknown attributes: raises `AttributeError`. This is intentional:
   `CatalogContentListingObject.__getattr__()` catches `AttributeError`
   and falls back to `getObject()`, loading the real content object.

docs/sources/reference/schema.md

@@ -118,7 +118,16 @@ together in a single JSONB document. Example for a typical Plone Page:
 
 Key conventions:
 
-- Dates are stored as ISO 8601 strings with timezone offsets.
+- **Index values** (used for PG queries) are converted to JSON-safe types:
+  dates become ISO 8601 strings, etc.  These live at the top level of idx.
+- **Metadata values** that are JSON-native (str, int, float, bool, None,
+  and lists/dicts of these) also live at the top level of idx.
+- **Non-JSON-native metadata** (Zope `DateTime`, stdlib `datetime`, `date`,
+  etc.) is encoded via the Rust codec (`zodb-json-codec`) into a nested
+  `"@meta"` key.  This preserves original Python types so that
+  `brain.effective` returns a `DateTime` object, not a string.
+  The `@meta` dict uses codec type markers (e.g. `@dt`, `@cls`+`@s`) and
+  is decoded once per brain on first access (cached thereafter).
 - Multi-value fields (e.g., `Subject`, `allowedRolesAndUsers`) are
   stored as JSON arrays.
 - Boolean fields are stored as JSON `true`/`false`.
@@ -127,6 +136,10 @@ Key conventions:
 - Path fields (`path`, `path_parent`, `path_depth`) are stored in
   both the dedicated table columns and the idx JSONB for unified
   path query support.
+- Fields that are both indexes and metadata (e.g. `effective`) appear
+  in both places: top-level idx holds the converted ISO string (for
+  PG queries), while `@meta` holds the original `DateTime` (for brain
+  attribute access).
 
 ## SQL Functions
 

docs/sources/reference/zcatalog-compat.md

@@ -92,11 +92,18 @@ index._index.get("Document")   # PG query returning matching ZOIDs
 
 `PGCatalogBrain` attribute access follows these rules:
 
-| Attribute Type | Behavior |
+| Attribute Type | Resolution Order |
 |---|---|
-| Known index or metadata (in `IndexRegistry`) | Returns value from `idx` JSONB, or `None` if missing |
+| Known index or metadata (in `IndexRegistry`) | 1. `idx["@meta"]` (codec-decoded, for non-JSON-native types like `DateTime`) → 2. top-level `idx` JSONB → 3. `None` if missing from both |
 | Unknown attribute | Raises `AttributeError` |
 
+Non-JSON-native metadata values (Zope `DateTime`, `datetime`, `date`,
+etc.) are stored under `idx["@meta"]` at write time via the Rust codec
+(`pickle_to_dict`).  On first access, the `@meta` dict is decoded via
+`dict_to_pickle` + `pickle.loads` and cached per brain for the lifetime
+of the result set.  This ensures `brain.effective` returns a `DateTime`
+object, not an ISO string.
+
 The `AttributeError` for unknown attributes is intentional:
 `CatalogContentListingObject.__getattr__()` catches it and falls back to
 `getObject()`, loading the real content object.  Returning `None` instead