Skip to content

Commit 3f080aa

Browse files
jensensjensens
authored
Merge pull request #10 from bluedynamics/fix/pbauer-bug-reports
Fix bug reports from @pbauer (GH #9) + example distribution
2 parents 2490786 + 4758f1b commit 3f080aa

25 files changed

Lines changed: 1603 additions & 87 deletions

‎ARCHITECTURE.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ All catalog data lives in these columns -- no BTree/Bucket objects are written t
2727
| `config.py` | `CatalogStateProcessor`, pool discovery, DRI translator registration |
2828
| `schema.py` | DDL for catalog columns, functions, and indexes |
2929
| `brain.py` | `PGCatalogBrain` + lazy `CatalogSearchResults` |
30+
| `pgindex.py` | `PGIndex`, `PGCatalogIndexes` -- ZCatalog internal API wrappers |
3031
| `backends.py` | `SearchBackend` ABC, `TsvectorBackend`, `BM25Backend` |
3132
| `dri.py` | `DateRecurringIndexTranslator` for recurring events |
3233
| `interfaces.py` | `IPGCatalogTool`, `IPGIndexTranslator` |
@@ -233,6 +234,36 @@ The active backend is a module-level singleton. `config.py` calls `detect_and_se
233234

234235
`_pending_brains_for_path()` scans the pending store, matches paths, and returns `_PendingBrain` instances with just enough interface (`getPath()`, `_unrestrictedGetObject()`) for `reindexObjectSecurity` to work.
235236

237+
## ZCatalog Internal API Compatibility
238+
239+
Plone code accesses ZCatalog internal data structures directly. Since PlonePGCatalogTool never populates ZCatalog's BTrees, these are replaced with PG-backed implementations.
240+
241+
### PGIndex Wrappers (`pgindex.py`)
242+
243+
`PGCatalogIndexes` (class attribute `Indexes` on `PlonePGCatalogTool`) overrides `ZCatalogIndexes._getOb()` to wrap each returned index with `PGIndex`. Special indexes with `idx_key=None` are returned unwrapped.
244+
245+
`PGIndex` proxies a real ZCatalog index object, delegating all standard methods via `__getattr__`. It overrides:
246+
247+
- **`_index`** (property): Returns a `_PGIndexMapping` that translates `_index.get(value)` into a PG query on `idx` JSONB, returning ZOID as the record ID. Used by `plone.app.uuid.uuidToPhysicalPath()`.
248+
- **`uniqueValues()`**: PG `SELECT DISTINCT` / `GROUP BY` on idx JSONB. Used by `plone.app.dexterity` and `plone.restapi`.
249+
250+
### getpath / getrid (`catalog.py`)
251+
252+
- **`getpath(rid)`**: `SELECT path FROM object_state WHERE zoid = %(rid)s`. Raises `KeyError` if not found (matching ZCatalog). Used by `plone.app.uuid`.
253+
- **`getrid(path)`**: `SELECT zoid FROM object_state WHERE path = %(path)s`. Returns `default` if not found. Used by `plone.app.vocabularies`.
254+
255+
ZOID serves as the record ID (RID), matching the integer PK in PostgreSQL.
256+
257+
### Brain Attribute Resolution (`brain.py`)
258+
259+
`PGCatalogBrain.__getattr__` uses `_resolve_from_idx()` to distinguish known catalog fields from unknown attributes:
260+
261+
- **In idx**: Return the value
262+
- **Known field** (in `IndexRegistry` indexes or metadata) but absent from idx: Return `None` (Missing Value behavior, matching ZCatalog)
263+
- **Unknown field**: Raise `AttributeError`
264+
265+
This enables `CatalogContentListingObject.__getattr__` to fall back to `getObject()` for non-catalog attributes (e.g. `content_type`), matching the behavior of ZCatalog's `AbstractCatalogBrain` (which inherits from `Record` and only knows schema-defined attributes).
266+
236267
## Query Translation
237268

238269
`query.py` translates ZCatalog query dicts into parameterized SQL.

‎CHANGES.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,49 @@
11
# Changelog
22

3+
## 1.0.0b7
4+
5+
### Fixed
6+
7+
- `sort_on` now accepts a list of index names for multi-column sorting,
8+
matching ZCatalog's API. `sort_order` can also be a list (one direction
9+
per sort key) or a single string applied to all keys.
10+
11+
- `PGCatalogBrain.__getattr__` now distinguishes known catalog fields from
12+
unknown attributes. Known indexes and metadata columns return `None` when
13+
absent from idx (matching ZCatalog's Missing Value behavior), while unknown
14+
attributes raise `AttributeError`. This enables
15+
`CatalogContentListingObject.__getattr__` to fall back to `getObject()`
16+
for non-catalog attributes (e.g. `content_type`), and fixes PAM's
17+
`get_alternate_languages()` viewlet crash on `brain.Language`.
18+
19+
- `reindexIndex` now accepts `pghandler` keyword argument for compatibility
20+
with ZCatalog's `manage_reindexIndex` and plone.distribution. The argument
21+
is accepted but ignored (PG-based reindexing doesn't need progress
22+
reporting). [#9]
23+
24+
- `clearFindAndRebuild` now properly rebuilds the catalog by traversing all
25+
content objects after clearing PG data. Previously only cleared without
26+
rebuilding.
27+
28+
- `refreshCatalog` now properly re-catalogs objects by resolving them from
29+
ZODB and re-extracting index values. Added missing `pghandler` parameter
30+
for ZCatalog API compatibility.
31+
32+
- Fixed `ConnectionStateError` on Zope restart when a Plone site already
33+
exists in the database. `_sync_registry_from_db` and
34+
`_detect_languages_from_db` now abort the transaction before closing
35+
their temporary ZODB connections.
36+
37+
- `_ensure_catalog_indexes` now checks for essential Plone indexes (UID,
38+
portal_type) instead of any indexes, preventing addon indexes from
39+
blocking re-application of Plone defaults.
40+
41+
- ZCatalog internal API compatibility: `getpath(rid)`, `getrid(path)`,
42+
`Indexes["UID"]._index.get(uuid)`, and `uniqueValues(withLengths=True)`
43+
now work with PG-backed data. Uses ZOID as the record ID. This fixes
44+
`plone.api.content.get(UID=...)`, `plone.app.vocabularies` content
45+
validation, and dexterity type counting in the control panel.
46+
347
## 1.0.0b6
448

549
### Added

‎README.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,11 +63,69 @@ results = catalog(start={
6363
})
6464
```
6565

66+
## Migrating an Existing Site
67+
68+
If you have a running Plone site and want to switch from ZCatalog to plone.pgcatalog:
69+
70+
**Prerequisites:** Your site must already be running on
71+
[zodb-pgjsonb](https://github.com/bluedynamics/zodb-pgjsonb).
72+
If you're migrating from FileStorage or RelStorage, use [zodb-convert](https://pypi.org/project/zodb-convert/) first.
73+
74+
**Steps:**
75+
76+
1. Install plone-pgcatalog into your Python environment:
77+
78+
```bash
79+
pip install plone-pgcatalog
80+
```
81+
82+
2. Restart Zope (plone.pgcatalog is auto-discovered via `z3c.autoinclude`).
83+
84+
3. Install the `plone.pgcatalog:default` GenericSetup profile -- either through the Plone Add-on control panel or programmatically:
85+
86+
```python
87+
setup = portal.portal_setup
88+
setup.runAllImportStepsFromProfile("profile-plone.pgcatalog:default")
89+
```
90+
91+
This replaces `portal_catalog` with `PlonePGCatalogTool` via `toolset.xml`.
92+
93+
4. Rebuild the catalog to populate PostgreSQL with all existing content:
94+
95+
```python
96+
catalog = portal.portal_catalog
97+
catalog.clearFindAndRebuild()
98+
```
99+
100+
For a site with ~1000 documents, this takes about 15 seconds.
101+
102+
An automated migration script is included in `example/scripts/migrate_to_pgcatalog.py`
103+
that performs all steps and verifies the result.
104+
105+
## Using with plone.distribution
106+
107+
An example distribution package is included in `example/pgcatalog-example-distribution/`.
108+
It registers a **"Plone Site (PG Catalog)"** distribution that appears in the site creation UI
109+
and automatically applies the `plone.pgcatalog:default` profile.
110+
111+
To use plone.pgcatalog in your own distribution, add it to `profiles.json`:
112+
113+
```json
114+
{
115+
"base": [
116+
"plone.app.contenttypes:default",
117+
"plonetheme.barceloneta:default",
118+
"plone.pgcatalog:default"
119+
]
120+
}
121+
```
122+
66123
## Documentation
67124

68125
- [ARCHITECTURE.md](ARCHITECTURE.md) -- internal design, index registry, query translation, custom index types
69126
- [BENCHMARKS.md](BENCHMARKS.md) -- performance comparison vs RelStorage+ZCatalog
70127
- [CHANGES.md](CHANGES.md) -- changelog
128+
- [example/](example/) -- runnable example with multilingual content and an example distribution
71129

72130
## Source Code and Contributions
73131

‎example/README.md‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -221,6 +221,88 @@ WHERE idx->'allowedRolesAndUsers' ?| ARRAY['Anonymous']
221221
ORDER BY path;
222222
```
223223

224+
## Example Distribution
225+
226+
The `pgcatalog-example-distribution/` directory contains a minimal
227+
[plone.distribution](https://github.com/plone/plone.distribution) package
228+
that registers a **"Plone Site (PG Catalog)"** distribution.
229+
230+
It is included in `requirements.txt` and auto-discovered by Plone at startup.
231+
232+
You can create a site via the REST API:
233+
234+
```bash
235+
curl -u admin:admin -X POST http://localhost:8081/@sites \
236+
-H "Accept: application/json" \
237+
-H "Content-Type: application/json" \
238+
-d '{
239+
"distribution": "pgcatalog_demo",
240+
"site_id": "demo",
241+
"title": "PG Catalog Demo",
242+
"default_language": "en",
243+
"portal_timezone": "Europe/Berlin",
244+
"setup_content": true
245+
}'
246+
```
247+
248+
Use this as a template for your own addon/distribution that depends on
249+
plone.pgcatalog -- the key is adding `plone.pgcatalog:default` to
250+
`profiles.json`.
251+
252+
## Migration Test
253+
254+
Test migrating an existing Plone site (standard ZCatalog) to plone.pgcatalog.
255+
256+
### 1. Create a plain site without pgcatalog
257+
258+
```bash
259+
# Remove plone-pgcatalog from the venv (if installed)
260+
uv pip uninstall plone-pgcatalog pgcatalog-example
261+
262+
# Create a Plone site with Wikipedia content, using standard ZCatalog
263+
.venv/bin/zconsole run instance/etc/zope.conf scripts/create_plain_site.py
264+
```
265+
266+
This creates `/Plone` with ~1000 multilingual Documents using the standard
267+
`CatalogTool` (ZCatalog BTrees). No pgcatalog involved.
268+
269+
### 2. Install pgcatalog and migrate
270+
271+
```bash
272+
# Install plone-pgcatalog back
273+
uv pip install -e ../.. # or: uv pip install plone-pgcatalog
274+
275+
# Run the migration script
276+
.venv/bin/zconsole run instance/etc/zope.conf scripts/migrate_to_pgcatalog.py
277+
```
278+
279+
The migration script:
280+
1. Installs the `plone.pgcatalog:default` GenericSetup profile (replaces
281+
`portal_catalog` with `PlonePGCatalogTool`)
282+
2. Runs `clearFindAndRebuild` to populate PostgreSQL from existing content
283+
3. Verifies all content is indexed and searchable
284+
285+
Expected output:
286+
```
287+
Before: catalog class = CatalogTool
288+
Before: 1072 objects indexed
289+
290+
Installing plone.pgcatalog:default profile ...
291+
After: catalog class = PlonePGCatalogTool
292+
After: 30 ZCatalog indexes registered
293+
294+
Rebuilding catalog (clearFindAndRebuild) ...
295+
Rebuild completed in 15.4s
296+
297+
Verification:
298+
Total indexed: 1072
299+
Documents: 1062
300+
SearchableText='volcano': 63 hits
301+
path=/Plone/en/library depth=1: 393 hits
302+
303+
Migration successful! All content is indexed and searchable.
304+
```
305+
224306
## Cleanup
225307

226308
```bash

‎example/pgcatalog-example-distribution/pyproject.toml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
[build-system]
2+
requires = ["setuptools>=68.2"]
3+
build-backend = "setuptools.build_meta"
4+
5+
[project]
6+
name = "pgcatalog-example"
7+
version = "1.0.0a1"
8+
description = "Example Plone distribution using plone-pgcatalog"
9+
requires-python = ">=3.12"
10+
dependencies = [
11+
"Products.CMFPlone",
12+
"plone.distribution",
13+
"plone-pgcatalog",
14+
]
15+
classifiers = [
16+
"Framework :: Plone",
17+
"Framework :: Plone :: 6.1",
18+
"Framework :: Plone :: Distribution",
19+
]
20+
21+
[tool.setuptools.packages.find]
22+
where = ["src"]
23+
24+
[tool.setuptools.package-data]
25+
pgcatalog_example = ["*.zcml", "distributions/**/*.json"]
26+
27+
[project.entry-points."z3c.autoinclude.plugin"]
28+
target = "plone"

‎example/pgcatalog-example-distribution/src/pgcatalog_example/__init__.py‎

Whitespace-only changes.

‎example/pgcatalog-example-distribution/src/pgcatalog_example/configure.zcml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
<configure xmlns="http://namespaces.zope.org/zope">
2+
3+
<include file="dependencies.zcml" />
4+
<include file="distributions.zcml" />
5+
6+
</configure>

‎example/pgcatalog-example-distribution/src/pgcatalog_example/dependencies.zcml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
<configure xmlns="http://namespaces.zope.org/zope">
2+
3+
<include package="Products.CMFPlone" />
4+
<include package="plone.distribution" />
5+
<include package="plone.pgcatalog" />
6+
7+
</configure>

‎example/pgcatalog-example-distribution/src/pgcatalog_example/distributions.zcml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
<configure
2+
xmlns="http://namespaces.zope.org/zope"
3+
xmlns:plone="http://namespaces.plone.org/plone"
4+
>
5+
6+
<plone:distribution
7+
name="pgcatalog_demo"
8+
title="Plone Site (PG Catalog)"
9+
description="A Plone Site using plone-pgcatalog for PostgreSQL-backed catalog."
10+
headless="false"
11+
post_handler="plone.distribution.handler.post_handler"
12+
/>
13+
14+
</configure>

‎example/pgcatalog-example-distribution/src/pgcatalog_example/distributions/pgcatalog_demo/profiles.json‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
{
2+
"base": [
3+
"plone.app.contenttypes:default",
4+
"plonetheme.barceloneta:default",
5+
"plone.pgcatalog:default"
6+
],
7+
"content": [
8+
]
9+
}

0 commit comments

Comments
 (0)