Skip to content

Add GrampsWebApiDb: use a Gramps Web API server as a live database backend - #1009

Open
dsblank wants to merge 11 commits into
gramps-project:maintenance/gramps60from
dsblank:add-grampswebapidb
Open

Add GrampsWebApiDb: use a Gramps Web API server as a live database backend#1009
dsblank wants to merge 11 commits into
gramps-project:maintenance/gramps60from
dsblank:add-grampswebapidb

Conversation

@dsblank

@dsblank dsblank commented Aug 4, 2026

Copy link
Copy Markdown
Member

Summary

GrampsWebApiDb is a DATABASE-type addon that lets Gramps (desktop or
scripted) open a gramps-web-api server (e.g. a gramps-connect or Gramps
Web instance) as a regular family tree — read and write, no
export/import step, no separate sync tool to run by hand.

It's not fast (every write is a network round trip), but it means the
same tree hosted online is simultaneously usable from Gramps desktop and
from the web app, live.

How it works

Rather than implementing DbReadBase/DbWriteBase from scratch (~175
methods), WebApiDB subclasses the stock SQLite DBAPI backend.
DbGeneric already implements every get_*/iter_*/commit_*/
remove_* method generically on top of a small SQL connection, so this
addon only needs to supply two things:

  • A local SQLite mirror for fast reads, kept in sync via
    GET /transactions/history/?after=<high-water-mark> — the same
    per-object transaction log the server's own undo system uses. The
    high-water-mark is persisted in the mirror's own metadata table
    (_get_metadata/_set_metadata), so reopening an existing tree is
    incremental, not a full re-download.
  • Write-through, hooked at transaction_commit() (the single point
    DbTxn.__exit__ calls after any completed local transaction) rather
    than each individual commit_person/commit_family/etc. Local edits
    push to POST /transactions/ automatically.

webapi_client.py is a small, self-contained HTTP client (stdlib
urllib only, no extra dependency) — trimmed from and crediting the
WebApiHandler class in the GrampsWebSync addon (same repo). It's now
a hand-synced vendored copy of a standalone gramps-api-client
package (not yet published; same code, class renamed Client) — the
addon keeps its own copy since Gramps addons can't declare a pip
dependency. That standalone package also ships a
gramps-api-client generate-key CLI as an alternative to the
mint_api_key() script-based flow described below.

Credentials

A single environment variable, GRAMPS_WEB_API_KEY, shaped
"<REFRESH_TOKEN>*<BASE64URL(URL)>". This is deliberately not a login
dialog: requires_login() returns False, and the same env var also
works as a bare SDK credential (WebApiHandler.from_env()) for scripts
that talk to the server directly, without Gramps involved at all — one
credential, two consumers.

The TOKEN half is a non-expiring JWT refresh token
(gramps-web-api's JWT_REFRESH_TOKEN_EXPIRES is False by default),
obtained once via WebApiHandler.mint_api_key(url, username, password)
(or the gramps-api-client generate-key CLI mentioned above). This
is a known, deliberate stopgap, not a real personal-access-token:
gramps-web-api has scoped, revocable token infrastructure already
(AccessToken, hashed, per-scope) but it's currently hardcoded to a
single scope (anniversaries_ics) and not wired into general request
auth. Until that's generalized server-side, this key is exactly as
powerful as the account it was minted from, and — unlike a real
personal-access-token — isn't individually revocable (a password change
doesn't invalidate it). This tradeoff and the reasoning are documented
in webapi_client.py's module docstring.

Usage

  1. Mint a GRAMPS_WEB_API_KEY -- it bakes in the host, and logs in as a
    particular user/database on that server. Via the CLI, for example:
$ gramps-api-client generate-key --url "http://localhost:5003/api" --user "gramps"
Password:...
<the key>
  1. export GRAMPS_WEB_API_KEY="<the key>"
  2. Add the addon in Gramps
  3. Start Gramps; in Preferences, change the database backend to "Gramps
    Web API Database".
image
  1. Create a new family tree. The type just reflects what the GRAMPS_WEB_API_KEY points to at that time.
image

Gramps opens the tree already populated with the server's data (the
initial sync), and from there it behaves like any other family tree:
edit, delete, undo, and redo all work, with every change pushed back to
the server live.

Status

  • ✅ Read sync (_sync_from_server), incremental, verified against a
    live server
  • ✅ Write-through (transaction_commit hook), verified add/update/delete
    round-trip against a live server, confirmed via an independent fresh
    mirror that pushes actually reach the server
  • ✅ Verified live in Gramps desktop itself (not just scripts): backend
    selectable in the New Family Tree dialog, syncs and edits correctly
  • ✅ Push conflict detection — pushes no longer go out with force=1;
    the server's optimistic-concurrency check (comparing each item's
    pre-edit snapshot against its current data) now actually runs, and a
    rejection raises WebApiPushConflict, which transaction_commit()
    catches to resync the local mirror from the server instead of
    silently overwriting. Real conflict resolution (merge, prompt the
    user) is still out of scope — the losing local edit is simply lost
    from the server's perspective.
  • ✅ Undo/redo integration — undo()/redo() are overridden to push to
    the server too, not just the local mirror: undo sends the original
    transaction's payload to POST /transactions/?undo=1 (the server
    reverses it itself), redo just re-pushes it forward, same as an
    ordinary commit. Both go through the same conflict-detection/resync
    path as a plain commit. Verified end-to-end against a live server
    (add a person, undo, confirm via a fresh mirror sync that the server
    no longer has it, redo, confirm it's back). Gramps' own undo history
    is in-memory/per-session, not persisted, so this only matters within
    a single running session.
  • ❌ No media file sync

Registered status=UNSTABLE to reflect the remaining gaps. Targets
gramps_target_version="6.0".

Testing

  • Automated: python3 -m unittest GrampsWebApiDb.tests.test_webapi_client GrampsWebApiDb.tests.test_grampswebapidb -v — 67 tests, covering
    webapi_client.py's auth/retry/rate-limit/conflict logic and
    grampswebapidb.py's sync/apply-change/transaction-commit/undo-redo
    logic, with urlopen and the SQLite base class mocked (no live server
    needed).
  • Manual, end-to-end against a real gramps-web-api server: read sync,
    write-through, incremental re-sync, push-conflict detection + resync,
    undo/redo (round-tripped through independent fresh mirrors to confirm
    the server's state, not just the local one), and live use from Gramps
    desktop (New Family Tree dialog, backend selectable, syncs and edits
    correctly). Also verified make_database("grampswebapidb") loads the
    addon through Gramps's real plugin manager, not just via direct
    import.

dsblank and others added 10 commits August 4, 2026 14:48
…ckend

Lets Gramps open a gramps-web-api server (e.g. gramps-connect, Gramps Web)
as a regular family tree -- read and write, no export/import step.
Subclasses the stock SQLite DBAPI backend rather than reimplementing
DbReadBase/DbWriteBase, and keeps a local mirror in sync incrementally via
the server's transaction-history endpoint; local edits push back through
transaction_commit(). Credentials come from a single GRAMPS_WEB_API_KEY
env var (a non-expiring refresh token) rather than a login dialog, which
also makes the same webapi_client.py usable as a bare SDK outside Gramps.

Status UNSTABLE: no conflict handling (writes are last-write-wins by
design, not yet), no undo/redo integration, no media sync. Verified
end-to-end against a live gramps-web-api server, including live use from
Gramps desktop itself, but no automated test suite yet.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…radeoff

Explains that GRAMPS_WEB_API_KEY carries a standard, non-expiring
refresh token from the server's normal login flow rather than a
scoped/revocable personal access token, so a leaked key is as
damaging as a leaked password until it's changed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Cover webapi_client.WebApiHandler (token codec, JWT decoding, auth flows,
429/401 retry and API-prefix fallback, transaction-history/push request
shape) and grampswebapidb.WebApiDB (transaction_to_json, _apply_change,
_sync_from_server pagination, transaction_commit ordering and error
handling). No real server or SQLite file is needed; urlopen and the
DBAPI/SQLite base are stubbed throughout.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Drop force=1 from POST /transactions/ so the server's old-data-mismatch
check actually runs. A rejected push now raises WebApiPushConflict
(webapi_client.py), which transaction_commit() catches separately from
generic connection errors: it logs a distinct warning and resyncs from
the server so the local mirror stops showing an edit the server never
accepted, rather than drifting silently. webapi_client.py's docstring
also now notes it's a hand-synced vendored copy of the standalone
gramps-web-api-client package.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…y CLI

Documents the standalone package's CLI as the primary way to mint a
GRAMPS_WEB_API_KEY, with the addon's own vendored
WebApiHandler.mint_api_key() as the equivalent no-extra-dependency
fallback.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…-api-client rename

The standalone client package was renamed (gramps_web_api_client ->
gramps_api_client, new checkout at ~/gramps/gramps-api-client). Updated
webapi_client.py's docstring and README.md accordingly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ncing it

Gramps core's DbGenericUndo._undo()/_redo() revert the local mirror via
low-level _txn_begin()/undo_data()/_txn_commit() calls that never go
through transaction_commit(), so a local Undo/Redo previously left the
server unchanged with no error at all -- worse than a push conflict,
since nothing was even logged.

WebApiDB now overrides undo()/redo(): both grab the relevant DbTxn off
DbGenericUndo's queue before delegating to super(), rebuild its payload
with the existing transaction_to_json(), and push it. Undo sends it to
POST /transactions/?undo=1, where gramps-web-api reverses it itself
(reverse_transaction()); redo just re-pushes the original forward
payload, same as an ordinary commit. Both share the same
conflict-detection/resync path as transaction_commit(), factored out
into _push_payload().

Verified end-to-end against a live server: add a person, undo (a fresh
mirror sync confirms the server no longer has it), redo (confirms it's
back).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- grampswebapidb.py: use the documented try/except get_addon_translator
  fallback (glocale.translation directly if the addon has no locale/
  translations yet) instead of a bare glocale.translation.gettext.
- grampswebapidb.gpr.py: add help_url pointing at the addon's wiki page.
- MANIFEST: include README.md in the built .addon.tgz -- it documents
  the GRAMPS_WEB_API_KEY security tradeoff, not just dev notes.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ommit gap

A batch=True commit (any bulk import, merge, or tool run through
gramps-web-api) leaves an empty-changes marker in the transaction
history instead of per-object entries, since DBAPI's commit/remove
methods skip the undo-log call for batch transactions. _sync_from_server()
had no way to detect this and silently missed everything the batch
commit did -- confirmed live: importing example.gramps's 2157 people
into a synced tree left the local mirror stuck at its pre-import count
indefinitely, no matter how often it resynced.

_sync_from_server() now treats an empty-changes transaction as a
signal, not a no-op, and falls back to a new _full_resync(): download
the server's current Gramps XML export and reimport it (via the same
stock ImportXml the batch commit itself used) into a freshly wiped
local mirror.
@dsblank
dsblank marked this pull request as ready for review August 5, 2026 02:17
…live

load() now schedules a periodic re-sync (GLib.timeout_add_seconds), matching
gramps-connect's own browser-client poll against the same endpoint, so edits
made from another client show up here without closing and reopening the
tree; close() cancels the pending timeout. Since _sync_from_server()'s
replay runs in a batch=True DbTxn, DBAPI's own add/update/delete signals
never fire for it -- _emit_change_signals() reproduces them by hand so
already-open views refresh the same way they would for a local edit, and
_full_resync() now calls request_rebuild() for the same reason on its
wipe-and-reimport path.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant