Add GrampsWebApiDb: use a Gramps Web API server as a live database backend - #1009
Open
dsblank wants to merge 11 commits into
Open
Add GrampsWebApiDb: use a Gramps Web API server as a live database backend#1009dsblank wants to merge 11 commits into
dsblank wants to merge 11 commits into
Conversation
…ckend Lets Gramps open a gramps-web-api server (e.g. gramps-connect, Gramps Web) as a regular family tree -- read and write, no export/import step. Subclasses the stock SQLite DBAPI backend rather than reimplementing DbReadBase/DbWriteBase, and keeps a local mirror in sync incrementally via the server's transaction-history endpoint; local edits push back through transaction_commit(). Credentials come from a single GRAMPS_WEB_API_KEY env var (a non-expiring refresh token) rather than a login dialog, which also makes the same webapi_client.py usable as a bare SDK outside Gramps. Status UNSTABLE: no conflict handling (writes are last-write-wins by design, not yet), no undo/redo integration, no media sync. Verified end-to-end against a live gramps-web-api server, including live use from Gramps desktop itself, but no automated test suite yet. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…radeoff Explains that GRAMPS_WEB_API_KEY carries a standard, non-expiring refresh token from the server's normal login flow rather than a scoped/revocable personal access token, so a leaked key is as damaging as a leaked password until it's changed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Cover webapi_client.WebApiHandler (token codec, JWT decoding, auth flows, 429/401 retry and API-prefix fallback, transaction-history/push request shape) and grampswebapidb.WebApiDB (transaction_to_json, _apply_change, _sync_from_server pagination, transaction_commit ordering and error handling). No real server or SQLite file is needed; urlopen and the DBAPI/SQLite base are stubbed throughout. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Drop force=1 from POST /transactions/ so the server's old-data-mismatch check actually runs. A rejected push now raises WebApiPushConflict (webapi_client.py), which transaction_commit() catches separately from generic connection errors: it logs a distinct warning and resyncs from the server so the local mirror stops showing an edit the server never accepted, rather than drifting silently. webapi_client.py's docstring also now notes it's a hand-synced vendored copy of the standalone gramps-web-api-client package. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…y CLI Documents the standalone package's CLI as the primary way to mint a GRAMPS_WEB_API_KEY, with the addon's own vendored WebApiHandler.mint_api_key() as the equivalent no-extra-dependency fallback. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…-api-client rename The standalone client package was renamed (gramps_web_api_client -> gramps_api_client, new checkout at ~/gramps/gramps-api-client). Updated webapi_client.py's docstring and README.md accordingly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ncing it Gramps core's DbGenericUndo._undo()/_redo() revert the local mirror via low-level _txn_begin()/undo_data()/_txn_commit() calls that never go through transaction_commit(), so a local Undo/Redo previously left the server unchanged with no error at all -- worse than a push conflict, since nothing was even logged. WebApiDB now overrides undo()/redo(): both grab the relevant DbTxn off DbGenericUndo's queue before delegating to super(), rebuild its payload with the existing transaction_to_json(), and push it. Undo sends it to POST /transactions/?undo=1, where gramps-web-api reverses it itself (reverse_transaction()); redo just re-pushes the original forward payload, same as an ordinary commit. Both share the same conflict-detection/resync path as transaction_commit(), factored out into _push_payload(). Verified end-to-end against a live server: add a person, undo (a fresh mirror sync confirms the server no longer has it), redo (confirms it's back). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- grampswebapidb.py: use the documented try/except get_addon_translator fallback (glocale.translation directly if the addon has no locale/ translations yet) instead of a bare glocale.translation.gettext. - grampswebapidb.gpr.py: add help_url pointing at the addon's wiki page. - MANIFEST: include README.md in the built .addon.tgz -- it documents the GRAMPS_WEB_API_KEY security tradeoff, not just dev notes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ommit gap A batch=True commit (any bulk import, merge, or tool run through gramps-web-api) leaves an empty-changes marker in the transaction history instead of per-object entries, since DBAPI's commit/remove methods skip the undo-log call for batch transactions. _sync_from_server() had no way to detect this and silently missed everything the batch commit did -- confirmed live: importing example.gramps's 2157 people into a synced tree left the local mirror stuck at its pre-import count indefinitely, no matter how often it resynced. _sync_from_server() now treats an empty-changes transaction as a signal, not a no-op, and falls back to a new _full_resync(): download the server's current Gramps XML export and reimport it (via the same stock ImportXml the batch commit itself used) into a freshly wiped local mirror.
dsblank
marked this pull request as ready for review
August 5, 2026 02:17
…live load() now schedules a periodic re-sync (GLib.timeout_add_seconds), matching gramps-connect's own browser-client poll against the same endpoint, so edits made from another client show up here without closing and reopening the tree; close() cancels the pending timeout. Since _sync_from_server()'s replay runs in a batch=True DbTxn, DBAPI's own add/update/delete signals never fire for it -- _emit_change_signals() reproduces them by hand so already-open views refresh the same way they would for a local edit, and _full_resync() now calls request_rebuild() for the same reason on its wipe-and-reimport path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GrampsWebApiDbis aDATABASE-type addon that lets Gramps (desktop orscripted) open a gramps-web-api server (e.g. a gramps-connect or Gramps
Web instance) as a regular family tree — read and write, no
export/import step, no separate sync tool to run by hand.
It's not fast (every write is a network round trip), but it means the
same tree hosted online is simultaneously usable from Gramps desktop and
from the web app, live.
How it works
Rather than implementing
DbReadBase/DbWriteBasefrom scratch (~175methods),
WebApiDBsubclasses the stockSQLiteDBAPI backend.DbGenericalready implements everyget_*/iter_*/commit_*/remove_*method generically on top of a small SQL connection, so thisaddon only needs to supply two things:
GET /transactions/history/?after=<high-water-mark>— the sameper-object transaction log the server's own undo system uses. The
high-water-mark is persisted in the mirror's own metadata table
(
_get_metadata/_set_metadata), so reopening an existing tree isincremental, not a full re-download.
transaction_commit()(the single pointDbTxn.__exit__calls after any completed local transaction) ratherthan each individual
commit_person/commit_family/etc. Local editspush to
POST /transactions/automatically.webapi_client.pyis a small, self-contained HTTP client (stdliburllibonly, no extra dependency) — trimmed from and crediting theWebApiHandlerclass in theGrampsWebSyncaddon (same repo). It's nowa hand-synced vendored copy of a standalone
gramps-api-clientpackage (not yet published; same code, class renamed
Client) — theaddon keeps its own copy since Gramps addons can't declare a pip
dependency. That standalone package also ships a
gramps-api-client generate-keyCLI as an alternative to themint_api_key()script-based flow described below.Credentials
A single environment variable,
GRAMPS_WEB_API_KEY, shaped"<REFRESH_TOKEN>*<BASE64URL(URL)>". This is deliberately not a logindialog:
requires_login()returnsFalse, and the same env var alsoworks as a bare SDK credential (
WebApiHandler.from_env()) for scriptsthat talk to the server directly, without Gramps involved at all — one
credential, two consumers.
The
TOKENhalf is a non-expiring JWT refresh token(
gramps-web-api'sJWT_REFRESH_TOKEN_EXPIRESisFalseby default),obtained once via
WebApiHandler.mint_api_key(url, username, password)(or the
gramps-api-client generate-keyCLI mentioned above). Thisis a known, deliberate stopgap, not a real personal-access-token:
gramps-web-apihas scoped, revocable token infrastructure already(
AccessToken, hashed, per-scope) but it's currently hardcoded to asingle scope (
anniversaries_ics) and not wired into general requestauth. Until that's generalized server-side, this key is exactly as
powerful as the account it was minted from, and — unlike a real
personal-access-token — isn't individually revocable (a password change
doesn't invalidate it). This tradeoff and the reasoning are documented
in
webapi_client.py's module docstring.Usage
GRAMPS_WEB_API_KEY-- it bakes in the host, and logs in as aparticular user/database on that server. Via the CLI, for example:
export GRAMPS_WEB_API_KEY="<the key>"Web API Database".
GRAMPS_WEB_API_KEYpoints to at that time.Gramps opens the tree already populated with the server's data (the
initial sync), and from there it behaves like any other family tree:
edit, delete, undo, and redo all work, with every change pushed back to
the server live.
Status
_sync_from_server), incremental, verified against alive server
transaction_commithook), verified add/update/deleteround-trip against a live server, confirmed via an independent fresh
mirror that pushes actually reach the server
selectable in the New Family Tree dialog, syncs and edits correctly
force=1;the server's optimistic-concurrency check (comparing each item's
pre-edit snapshot against its current data) now actually runs, and a
rejection raises
WebApiPushConflict, whichtransaction_commit()catches to resync the local mirror from the server instead of
silently overwriting. Real conflict resolution (merge, prompt the
user) is still out of scope — the losing local edit is simply lost
from the server's perspective.
undo()/redo()are overridden to push tothe server too, not just the local mirror: undo sends the original
transaction's payload to
POST /transactions/?undo=1(the serverreverses it itself), redo just re-pushes it forward, same as an
ordinary commit. Both go through the same conflict-detection/resync
path as a plain commit. Verified end-to-end against a live server
(add a person, undo, confirm via a fresh mirror sync that the server
no longer has it, redo, confirm it's back). Gramps' own undo history
is in-memory/per-session, not persisted, so this only matters within
a single running session.
Registered
status=UNSTABLEto reflect the remaining gaps. Targetsgramps_target_version="6.0".Testing
python3 -m unittest GrampsWebApiDb.tests.test_webapi_client GrampsWebApiDb.tests.test_grampswebapidb -v— 67 tests, coveringwebapi_client.py's auth/retry/rate-limit/conflict logic andgrampswebapidb.py's sync/apply-change/transaction-commit/undo-redologic, with
urlopenand the SQLite base class mocked (no live serverneeded).
write-through, incremental re-sync, push-conflict detection + resync,
undo/redo (round-tripped through independent fresh mirrors to confirm
the server's state, not just the local one), and live use from Gramps
desktop (New Family Tree dialog, backend selectable, syncs and edits
correctly). Also verified
make_database("grampswebapidb")loads theaddon through Gramps's real plugin manager, not just via direct
import.