diff --git a/CHANGELOG.md b/CHANGELOG.md index 483db67c..7b1b37a5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,16 @@ Pre-1.0 note: while `pg_durable` is in major version `0`, minor releases may inc ### Added +- **Multi-database installations:** explicitly install in `pg_durable.database` + first, then add satellites with local `df` APIs, metadata, variables, grants, + and RLS. All installations share one control runtime/provider; satellites create + no `_duroxide` schema. SQL defaults to its origin unless an explicit target is + supplied. Public eight-character IDs and recorded payloads are unchanged; + satellite engine IDs carry database OID and installation UUID for activity routing. +- **Bounded origin connections:** `pg_durable.max_origin_connections` defaults to + `12` (range `2` to `1000`, restart required), shared by activities and maintenance. + Active routes reserve two slots; idle satellites retain no origin pool. + - **Failure-isolated loops:** the unified `df.loop(body, condition DEFAULT NULL, continue_on_failure DEFAULT false)` signature supports resilient infinite and conditional loops. With @@ -22,6 +32,16 @@ Pre-1.0 note: while `pg_durable` is in major version `0`, minor releases may inc ### Changed +- **Satellite lifecycle:** active activities hold installation/metadata locks so + drop waits for active operations; UUID fencing prevents stale execution after + recreation. Satellite engine cleanup is eventual on reconciliation, and failed + connections are not treated as proof of removal. Dropping control destroys + shared engine state for every satellite. +- **Independent-start scope:** `transaction_mode => 'new'` launches in the caller's + database, with `max_new_transaction_starts` enforced per database, not cluster-wide. +- **Shared metrics:** `df.metrics()` exposes all-engine totals even from satellites. + Granting `with_grant => true` includes this access as well as local delegation. + - **Loop lifetime:** raises the loop-iteration backstop from 100,000 to 8,388,608 (`2^23`), approximately 80 years at five-minute ticks. diff --git a/USER_GUIDE.md b/USER_GUIDE.md index 8dc3ce00..56774b49 100644 --- a/USER_GUIDE.md +++ b/USER_GUIDE.md @@ -74,10 +74,13 @@ pg_durable requires: available channels and their prerequisites. 2. **PostgreSQL configuration**: Add `pg_durable` to `shared_preload_libraries` in `postgresql.conf` 3. **Server restart**: Required after modifying `shared_preload_libraries` -4. **Extension creation**: Run `CREATE EXTENSION pg_durable` in your database +4. **Extension creation**: Install first in the control database named by + `pg_durable.database` (default `postgres`), then in any satellite databases ### Enable the Extension +Connect to the control database first: + ```sql CREATE EXTENSION pg_durable; @@ -85,9 +88,16 @@ CREATE EXTENSION pg_durable; SELECT df.grant_usage('app_role'); ``` -After `CREATE EXTENSION`, the background worker initializes the engine schema asynchronously (normally within a few seconds). Until initialization completes, `df.*` functions will return: `"pg_durable background worker not yet initialized — try again in a moment"`. Simply retry after a short delay. +After control `CREATE EXTENSION` commits, the background worker initializes the +engine asynchronously (normally within a few seconds). Operations needing the +engine report that it is not yet initialized until readiness is published. Once +control is ready, run `CREATE EXTENSION pg_durable` and local usage grants in each +satellite. Satellite installation checks control compatibility/readiness over SQLx +and fails if control is unavailable. See [Multi-Database Support](#multi-database-support). -> ⚠️ **Important**: If you include `pg_durable` in `shared_preload_libraries` but don't create the extension, the worker will remain idle and durable functions cannot execute. +> **Important:** Preloading alone does not create provider objects or execute +> workflows. Worker initialization and management/polling connections may still +> consume resources while waiting for explicit control installation. ### Your First Durable Function @@ -133,7 +143,8 @@ SELECT df.result('a1b2c3d4'); ### Instance IDs -Every durable function gets a unique 8-character hex ID (e.g., `a1b2c3d4`). Use this ID to: +Every durable function gets an 8-character hex ID (e.g., `a1b2c3d4`), unique within +its local installation. Use it in the database where the function was started to: - Check status: `SELECT df.status('a1b2c3d4')` - Get result: `SELECT df.result('a1b2c3d4')` - Cancel: `SELECT df.cancel('a1b2c3d4')` @@ -210,7 +221,8 @@ the default, so a typo cannot silently produce a start that does not survive the rollback you expected it to. Under `'new'` the start runs under the calling role's identity and privileges, -just as `'caller'` does. Three consequences follow from the separate session: +just as `'caller'` does. The loopback session connects to the caller's database, +even when `database` names a different SQL target. Consequences of the separate session: - **Variables are the committed snapshot.** The captured `df.vars` snapshot contains only *committed* variables. A `df.setvar()` issued earlier in the @@ -218,7 +230,7 @@ just as `'caller'` does. Three consequences follow from the separate session: calling, or pass values inline. - **It costs an extra backend.** Each call opens (and closes) a PostgreSQL connection, so it counts against `max_connections`. pg_durable caps these - extra loopback sessions cluster-wide with + extra loopback sessions **per database** using advisory admission locks and `pg_durable.max_new_transaction_starts` (default `2`); additional callers wait up to `pg_durable.new_transaction_start_timeout` seconds (default `5`) before failing without opening a second backend. Prefer the default on hot @@ -1538,7 +1550,23 @@ SELECT df.start( ## Multi-Database Support -By default, all SQL in a durable function runs in the database where the extension is installed (the `pg_durable.database` GUC, typically `postgres`). You can target a different database on the same cluster by passing the `database` parameter to `df.start()`. +pg_durable supports local installations in multiple databases with one shared +runtime. Distinguish three roles: + +- **Control database:** selected by `pg_durable.database` (default `postgres`), + with the single runtime/provider store. Install the extension here explicitly + and wait for worker readiness before installing satellites. +- **Origin database:** where `df.start()` is called. Each satellite has local + `df` APIs, instances, nodes, variables, grants, and RLS, but no `_duroxide` schema. +- **Execution database:** all SQL nodes use the origin by default. An explicit + `database` argument selects another database on the same cluster; that target + does not need the extension unless the SQL itself calls `df` APIs. + +Satellite installation requires a compatible control worker with readiness schema +version `2` (including its `_origins` registry). Provider migrations run only in +control through the worker's `ApplyAll`; satellite creation never creates control +automatically. The provider namespace is extension-owned, its contents are created +by `pg_durable.worker_role`, and satellite objects belong to the local installer. ### Running SQL in Another Database @@ -1562,9 +1590,35 @@ All SQL nodes in the function execute against the specified database. The DSL it ### Key Points - **One database per invocation.** All SQL in a single `df.start()` call targets the same database. For cross-database workflows, start separate durable functions per database, or use `dblink`/`postgres_fdw` within your SQL. -- **Backwards compatible.** Omitting `database` (or passing NULL) uses the extension database — existing queries are unaffected. +- **Origin-local defaults.** Omitting `database` (or passing NULL) uses the origin through activity routing; control-database behavior is unchanged. - **Validated at submission time.** If the database doesn't exist, `df.start()` raises an immediate error. - **Role isolation preserved.** The function runs as the user who called `df.start()`, not the background worker. The login role must be able to connect to the target database (`GRANT CONNECT`). +- **Local APIs and variables.** Use status, result, explain, signal, cancel, await, and listing APIs in the origin. RLS and variable capture remain local. Public IDs stay eight characters; the engine privately namespaces satellite IDs by database OID and installation UUID. +- **Local transactions.** Caller-mode starts retain local commit/rollback semantics. New-transaction starts launch in the caller's database and use a per-database admission cap. There is no cross-database atomicity between metadata, engine state, and SQL targets. + +### Worker Access and Lifecycle + +The default `pg_durable.worker_role` is the `postgres` superuser. A custom worker +role needs `CONNECT` and the required local `df` metadata/guard privileges in each +origin, plus access for origin-local HTTP privilege lookup. `BYPASSRLS` alone +grants none of these privileges. Application grants, including HTTP access, are +also database-local. A satellite admin granted `with_grant => true` can read +`df.metrics()` totals for **all** users and origins in the shared engine. + +Dropping a satellite destroys only its work. Active activities hold `ACCESS SHARE` +locks on `df._installation`, `df.instances`, and `df.nodes`, so drop waits for +active operations, not all queued or sleeping instances. The UUID check prevents +old work, including cached graphs, from executing after drop/recreate. Engine +cleanup is eventual on reconciliation; an unreachable origin is not treated as +absent. There is no DDL hook or cross-database reference count. + +**Dropping control destroys the shared engine for every satellite.** Local +satellite metadata cannot recover that history. Remove satellites first, control +last. Origin connections are bounded by +[`pg_durable.max_origin_connections`](docs/api-reference.md#pg_durablemax_origin_connections), +not by a ceiling on database names; idle satellites have no retained origin pool. +See [Multi-Database Extension Installation](docs/multi-database-installation.md) +for design and upgrade details. ### Example: Multi-Tenant Processing @@ -1952,13 +2006,18 @@ This is useful for dashboards and operational queries that need to understand wh ### System Metrics (Explicit Grant Required) ```sql --- Requires a direct admin grant; df.grant_usage() does not include it. +-- Requires an admin grant, including df.grant_usage(..., with_grant => true). SELECT * FROM df.metrics(); ``` **Columns:** `total_instances`, `running_instances`, `completed_instances`, `failed_instances`, `total_executions`, `total_events` -> **Note:** `df.metrics()` returns system-wide aggregate counts across all users and is omitted from an ordinary `df.grant_usage('role')`. It is granted automatically to pg_durable admins via `df.grant_usage('role', with_grant => true)`, or you can grant EXECUTE on `df.metrics()` directly to any role that may view cluster-wide pg_durable activity. Other users can call `df.list_instances()` to view a summary of their own workflows. +> **Note:** `df.metrics()` returns aggregate totals for the shared engine across +> all users and origins, even when called from a satellite. Ordinary +> `df.grant_usage('role')` omits it; `with_grant => true` grants it automatically +> along with local delegation rights. Granting satellite administration therefore +> also exposes all-engine totals. A direct `EXECUTE` grant is another explicit +> opt-in. `df.list_instances()` remains local and RLS-scoped. ### Quick Status Check @@ -1985,7 +2044,7 @@ If you reuse a label across runs, multiple instances can match — pass the spec ### Worker Liveness -Check whether the background worker is alive and healthy: +In the **control database**, check whether the background worker is alive and healthy: ```sql SELECT started_at, last_seen_at, @@ -1996,7 +2055,8 @@ SELECT started_at, last_seen_at, - `time_since_last_heartbeat < 15 seconds` → worker is alive (recent heartbeat) - No rows in `df._worker_epoch` → worker hasn't initialized yet -The background worker updates `last_seen_at` every ~5 seconds as part of its normal operation. +The background worker updates `last_seen_at` every ~5 seconds in control. An empty +satellite `df._worker_epoch` is normal; satellites do not have separate workers. ### Automatic Reconciliation @@ -2007,8 +2067,8 @@ worker runs a **best-effort reconciliation pass** that does two things: - **Removes expired terminal instances.** Old **terminal** instances (status `completed`, `failed`, or `cancelled`) and their `df.nodes` rows are deleted, - along with their engine records. Running and pending instances are **never** - removed, regardless of age. + along with their engine records. Retention does not remove running or pending + instances from an existing installation, regardless of age. - **Reclaims orphaned engine records.** `df.start()` writes the `df` rows in the caller's transaction but hands the workflow to the engine over a separate connection; if that transaction **rolls back**, the `df` rows vanish while the @@ -2017,6 +2077,16 @@ worker runs a **best-effort reconciliation pass** that does two things: age past `retention_days`. Anything the engine is still tracking with a live `df` row is left untouched. +Satellite maintenance visits `_origins` registrations made by activity routing for +submitted work, in bounded batches under the shared origin connection budget. +Registration is not synchronous with `df.start()`; unused installations are not +discovered by an all-database scan. After confirmed removal or replacement, +bounded reconciliation cancels running roots without waiting for retention; +terminal engine deletion still respects retention. Connection failures defer +cleanup rather than establish absence. Control removal is destructive to every +origin's engine state. Retention resumes from a bounded cursor so undeletable +engine records do not permanently block later candidates. + Two Postmaster-context GUCs govern it (set in `postgresql.conf`, restart to apply): ```ini @@ -2031,7 +2101,7 @@ pg_durable.retention_days = 30 Retention combines that window with a fixed hard cap: -- **Hard cap — at most 10,000 terminal instances are retained, regardless of +- **Hard cap per origin — at most 10,000 terminal instances are retained, regardless of age.** The newest 10,000 terminal instances are kept; any beyond that are removed even if they are only minutes old. (This cap is fixed, not a GUC.) - **Retention window — `retention_days`.** Terminal instances older than the @@ -2223,7 +2293,9 @@ own statement logging. Keep credentials out of SQL even when this setting is off ### Privilege Grants -`CREATE EXTENSION pg_durable` does **not** grant privileges to `PUBLIC`. After installing the extension, the admin must explicitly grant access to each application role. RLS ensures per-user isolation even when multiple roles share the same grants. +`CREATE EXTENSION pg_durable` does **not** grant general `df` usage to `PUBLIC`. +After installing, an admin must grant application access in each origin database. +RLS ensures per-user isolation even when multiple roles share the same local grants. **Recommended — use the built-in helper:** @@ -2242,7 +2314,11 @@ SELECT df.grant_usage('admin_role', include_http => true, with_grant => true); This function is purely additive — it never issues REVOKE. To downgrade a role's privileges (e.g., remove HTTP access), call `df.revoke_usage()` first, then `df.grant_usage()` with the desired options. -> **Granting to `PUBLIC`:** `df.grant_usage('public')` is allowed and grants `df` access to **every role in the cluster**, defeating the deny-by-default posture that a fresh install sets up. This is a deliberate, visible action (the same as any `GRANT ... TO PUBLIC`), not a mistake the helper blocks — use it only when you intend cluster-wide access. Naming a role that doesn't exist fails naturally on the first `GRANT`. +> **Granting to `PUBLIC`:** `df.grant_usage('public')` grants every role access to +> this database's `df` installation, not other installations. This defeats the +> local deny-by-default posture. Combining it with `with_grant => true` also +> exposes shared-engine metrics and delegation rights to every role. Naming a +> role that does not exist fails on the first `GRANT`. **Parameters:** @@ -2250,7 +2326,7 @@ This function is purely additive — it never issues REVOKE. To downgrade a role |-----------|---------|-------------| | `p_role` | *(required)* | Target role name | | `include_http` | `false` | Grant EXECUTE on `df.http()` (opt-in — makes outbound network requests) | -| `with_grant` | `false` | Grant all privileges WITH GRANT OPTION and allow the role to call `df.grant_usage()` / `df.revoke_usage()` to manage other roles' access. Also grants EXECUTE on `df.metrics()` (system-wide aggregate counts), since `with_grant => true` designates a pg_durable admin. The caller must hold each underlying privilege WITH GRANT OPTION (automatically true for superusers and delegated admins). | +| `with_grant` | `false` | Grant local privileges WITH GRANT OPTION and allow `df.grant_usage()` / `df.revoke_usage()` delegation. Also grants `df.metrics()`, exposing all-engine totals across users and origins even from a satellite. The caller must hold each underlying privilege WITH GRANT OPTION. |
Equivalent manual grants (for reference) @@ -2363,22 +2439,27 @@ This postmaster setting requires a PostgreSQL restart. When it is empty or unset ## Connection Limits -pg_durable uses multiple PostgreSQL connections for different purposes. Four GUCs let you control the connection budget to match your deployment's resources. +pg_durable uses separate limits for control management, provider, origin metadata, +SQL execution, and independent-start connections. ### Connection Architecture -The background worker maintains three categories of connections, and -`transaction_mode => 'new'` can transiently add a fourth category while starts -are being launched: +The main connection categories are: | Category | Purpose | GUC | Default | |----------|---------|-----|---------| -| **Management pool** | Extension lifecycle checks, graph loading, status updates | `pg_durable.max_management_connections` | 6 | +| **Management pool** | Control metadata, origin lookup and registration | `pg_durable.max_management_connections` | 6 | +| **Origin routes** | Satellite guard and metadata connections, shared by activities and maintenance | `pg_durable.max_origin_connections` | 12 | | **Duroxide pool** | Orchestration state, LISTEN/NOTIFY for work dispatch | `pg_durable.max_duroxide_connections` | 10 | | **User-execution** | Per-SQL-node connections authenticated as the submitting user | `pg_durable.max_user_connections` | 10 | -| **New-start loopback** | Extra sessions that persist `df.start(..., transaction_mode => 'new')` outside the caller's transaction | `pg_durable.max_new_transaction_starts` | 2 | +| **New-start loopback** | Extra launch sessions in the caller's database, capped per database | `pg_durable.max_new_transaction_starts` | 2 | -Each PG backend session (user calling `df.start()`, `df.cancel()`, etc.) creates **1 additional connection** for duroxide client operations. +The worker also has a dedicated one-connection control polling pool. Backend API +calls need additional control-client connections and one cached control-state +connection per satellite backend that uses the engine. Control-local readiness +uses SPI in the caller's transaction. Each active satellite +route reserves two origin slots (guard plus metadata); routes close afterward. +There is no idle pool per satellite or fixed database-name count ceiling. ### GUC Reference @@ -2387,10 +2468,14 @@ All connection-limit GUCs are **Postmaster-context** — set them in `postgresql ```ini # postgresql.conf -# Management pool: graph loading, status updates, lifecycle polling +# Control management pool: metadata, origin lookup and registration # Minimum: 1 (warning logged). Increase for high-concurrency workloads. pg_durable.max_management_connections = 6 +# Shared satellite connection budget for activities and maintenance. +# Range: 2..1000. Each active route reserves 2 slots (guard + metadata). +pg_durable.max_origin_connections = 12 + # Duroxide provider pool: orchestration state + LISTEN/NOTIFY # Minimum: 2 (1 reserved for listener). Worker refuses to start if < 2. pg_durable.max_duroxide_connections = 10 @@ -2403,7 +2488,7 @@ pg_durable.max_user_connections = 10 # before failing with an error. pg_durable.execution_acquire_timeout = 30 -# Maximum concurrent transaction_mode => 'new' loopback launch sessions. +# Maximum concurrent transaction_mode => 'new' launch sessions PER DATABASE. # Additional callers wait for a slot instead of opening more backends. pg_durable.max_new_transaction_starts = 2 @@ -2416,22 +2501,38 @@ pg_durable.new_transaction_start_timeout = 5 ### Connection Budget Formula -To calculate the total connections pg_durable will use: +Budget the worker's configured ceilings separately from backend API calls and +per-database launch sessions: ``` -Total = max_management_connections +Worker ceiling = max_management_connections + + max_origin_connections + max_duroxide_connections + max_user_connections - + max_new_transaction_starts - + (active_backend_sessions × 1) + + 1 (dedicated polling connection) + +Additional = sum of active new-start loopbacks across databases + + backend control-client/readiness connections ``` -With defaults and 5 connected users: `6 + 10 + 10 + 2 + 5 = 33 connections`. +The default worker ceiling is `6 + 12 + 10 + 10 + 1 = 39`, not an idle allocation. +Each database can additionally admit up to two new-start loopbacks by default. +Allow headroom for backend API connections as well as ordinary application sessions. > **Tip**: Ensure PostgreSQL's `max_connections` is large enough to accommodate pg_durable's budget plus your application's direct connections. ### Backpressure Behavior +Origin routes and maintenance share a semaphore. Admission waits up to 30 seconds +for origin slots before returning an error. Increasing +`max_management_connections` does not raise this satellite budget. + +Origin metadata queries have a 1.5-second lock timeout and a 5-second statement +timeout. These limits bound metadata contention and cancellation cleanup; they +do not limit user SQL or HTTP duration. Remote control-state probes have 1.5-second +server deadlines and a 5-second overall deadline. An unavailable control +installation returns an error instead of reusing stale readiness. + When all user-execution slots are occupied, additional SQL node executions **queue** (they don't fail immediately). The semaphore-based backpressure ensures: - Queued executions proceed as slots free up @@ -2447,7 +2548,7 @@ For `df.start(..., transaction_mode => 'new')`, admission control applies *before* the loopback session is opened: - At most `pg_durable.max_new_transaction_starts` loopback launch sessions exist - at once (default `2`) + at once **per database** (default `2`), enforced with advisory locks - Extra callers wait up to `pg_durable.new_transaction_start_timeout` seconds (default `5`) for a slot - If the wait expires, `df.start()` raises: @@ -2464,6 +2565,7 @@ The background worker validates GUC values at startup: - `max_duroxide_connections < 2` → worker **refuses to start** (logs error and exits) - `max_management_connections = 1` → worker starts but logs a **warning** +- `max_origin_connections` accepts `2` through `1000` (default `12`) - Invalid values are caught before any connections are created ### Interaction with PostgreSQL CONNECTION LIMIT @@ -2471,7 +2573,7 @@ The background worker validates GUC values at startup: PostgreSQL's per-role `CONNECTION LIMIT` (set via `ALTER ROLE ... CONNECTION LIMIT n`) counts against the **authenticating role** (the role in the connection string), not the role set via `SET ROLE`. For pg_durable, this means: -- **Management and duroxide pools** authenticate as `pg_durable.worker_role` — all pool connections count against that role's limit +- **Management, origin, polling, and duroxide pools** authenticate as `pg_durable.worker_role`; their connections count against that role's limit where PostgreSQL enforces it - **User-execution connections** authenticate as the submitting user (`submitted_by`) — these count against *that* role's limit - **Backend connections** authenticate as whatever role the application uses @@ -2484,12 +2586,13 @@ If you use per-role connection limits, ensure each role's limit accounts for pg_ pg_durable.max_management_connections = 3 pg_durable.max_duroxide_connections = 5 pg_durable.max_user_connections = 5 -# Budget: 3 + 5 + 5 + backends ≈ 15 connections +# Worker ceiling including default origin budget and polling: 3 + 12 + 5 + 5 + 1 = 26 +# Add per-database loopbacks and backend API/application connections. ``` **Medium deployment** (defaults — suitable for most workloads): ```ini -# Use defaults: 6 + 10 + 10 + backends ≈ 28 connections +# Default worker ceiling: 39; add loopbacks and backend API/application connections. ``` **Large deployment** (high concurrency, many parallel workflows): @@ -2498,7 +2601,8 @@ pg_durable.max_management_connections = 10 pg_durable.max_duroxide_connections = 15 pg_durable.max_user_connections = 50 pg_durable.execution_acquire_timeout = 60 -# Budget: 10 + 15 + 50 + backends ≈ 80 connections +# Worker ceiling including default origin budget and polling: 10 + 12 + 15 + 50 + 1 = 88 +# Add per-database loopbacks and backend API/application connections. ``` --- @@ -2537,7 +2641,7 @@ Failed to connect to duroxide store: ... **Possible Causes**: -1. **Extension not created**: Run `CREATE EXTENSION pg_durable` +1. **Control extension unavailable**: Install explicitly in `pg_durable.database` first, then in each satellite after control is ready. Recreating dropped control does not recover lost engine history. 2. **Background worker not yet ready**: After `CREATE EXTENSION`, the background worker initializes the engine schema asynchronously (normally within a few seconds). Simply retry after a short delay — once the worker finishes, the error resolves on its own. @@ -2555,31 +2659,31 @@ pg_durable: waiting for CREATE EXTENSION pg_durable... **Cause**: The background worker is waiting for the extension to be created in the database it's connected to. **Solution**: -1. Verify you're creating the extension in the correct database +1. Verify the control extension exists in `pg_durable.database` 2. Check which database the background worker connects to: - Controlled by the `pg_durable.database` GUC (set in `postgresql.conf`); defaults to `postgres` - - The background worker only processes functions in **one** database + - One runtime serves control and all registered satellite origins 3. If you need pg_durable in a different database: - - Create the extension in the database the background worker uses, OR - - Update `pg_durable.database` in `postgresql.conf` and restart PostgreSQL + - Wait for control readiness, then create a local satellite installation + - Verify worker-role access in both control and the satellite; changing + `pg_durable.database` is not a migration of existing engine state ### Extension Drop/Recreate Issues **Symptom**: After `DROP EXTENSION pg_durable CASCADE`, workflows still appear to be running or you see errors. -**Explanation**: The background worker polls for extension existence every 5 seconds. After detecting a drop: -- It shuts down the duroxide runtime (takes ~10 seconds) -- Returns to waiting for extension creation -- Any in-flight workflows are terminated - -> ⚠️ **`CASCADE` is always required.** The duroxide schema contains tables and functions created by the background worker that are not directly owned by the extension. `DROP EXTENSION pg_durable` (without `CASCADE`) will fail with an error. Always use `DROP EXTENSION pg_durable CASCADE`. - -**Solution**: Wait 15-20 seconds after `DROP EXTENSION` before recreating: -```sql -DROP EXTENSION pg_durable CASCADE; --- Wait ~20 seconds for background worker to fully shut down -CREATE EXTENSION pg_durable; -``` +**Satellite drop:** Active activities hold metadata/installation locks, so drop +waits for those operations. Once dropped, the UUID fence prevents old work from +executing after recreation, but its engine records can remain until reconciliation. +The shared runtime and other origins continue. Satellites have no provider objects; +`CASCADE` is needed only if other local dependencies require it. + +**Control drop:** The worker polls control existence about every five seconds and +tears down the runtime after detecting removal. Worker-created objects depend on +the extension-owned provider namespace, so control removal requires `CASCADE`. +This destroys engine history for every satellite. Do not use drop/recreate as an +upgrade procedure; use `ALTER EXTENSION UPDATE`. After intentional recreation, +wait for control readiness before starting new work or installing satellites. ### Functions Complete But Results Are Empty diff --git a/docs/api-reference.md b/docs/api-reference.md index 325c5f50..85a47c45 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -369,6 +369,12 @@ Returns the same envelope as `df.http()`. ## Control Functions +Instance APIs use eight-character local IDs in the database where the workflow +was started. Status, result, explain, signal, cancel, await, and listing operations +authorize against local `df` metadata/RLS, while engine operations use the shared +control store. Variables and HTTP grants are also origin-local. Satellite engine +IDs are internally namespaced; callers continue to pass the short local ID. + ### df.start(fut [, label] [, database] [, transaction_mode]) Starts a durable function. @@ -377,7 +383,7 @@ Starts a durable function. |-----------|------|-----------|-------------| | `fut` | TEXT | ✅ Auto-wrap | Root node of the function | | `label` | TEXT | ❌ Literal | (Optional) Human-readable label | -| `database` | TEXT | ❌ Literal | (Optional) Target database on the cluster | +| `database` | TEXT | ❌ Literal | (Optional) SQL target on the cluster; omitted/NULL defaults to the origin database where `df.start()` is called | | `transaction_mode` | TEXT | ❌ Literal | (Optional) `'caller'` (default) or `'new'` | ```sql @@ -386,6 +392,13 @@ df.start(df.sleep(10) ~> 'SELECT 2') -- explicit nodes df.start('SELECT 1', 'my-job') -- with label ``` +All SQL nodes share the selected execution database; metadata and captured +variables stay in the origin. The target does not need pg_durable unless the SQL +uses its APIs. SQL executes as captured `current_user`, with that role's target +database privileges. Install in `pg_durable.database` first and wait for control +readiness before creating satellites; see +[Multi-Database Support](../USER_GUIDE.md#multi-database-support). + #### transaction_mode Selects which transaction the *start itself* runs in. It changes nothing about @@ -394,7 +407,8 @@ the durable function that gets started. - `'caller'` (default) — the start joins the caller's transaction, so a `ROLLBACK` discards the durable function along with everything else. - `'new'` — the start runs in its own transaction on a separate PostgreSQL - session, so it commits independently and **survives a rollback of the + session in the caller's database, regardless of the SQL target. It commits + independently and **survives a rollback of the caller's transaction**. This provides the same rollback-survival outcome as an Oracle autonomous transaction for asynchronously started work. It is not a synchronous autonomous routine: the returned ID confirms the launch, while @@ -413,7 +427,8 @@ An unrecognised value raises an error rather than falling back to the default. > **Note:** under `'new'` the separate session sees only *committed* rows, so > the captured `df.vars` snapshot excludes variables set earlier in the caller's > open transaction. Each admitted call also opens an extra backend connection, -> capped cluster-wide by `pg_durable.max_new_transaction_starts` (default `2`); +> capped **per database** by advisory admission locks using +> `pg_durable.max_new_transaction_starts` (default `2`); > extra callers wait up to `pg_durable.new_transaction_start_timeout` seconds > (default `5`) before failing without opening the loopback session. The inner > `df.start()` statement itself is still bounded by a 30 s @@ -422,7 +437,8 @@ An unrecognised value raises an error rather than falling back to the default. > Avoid per-row triggers and other high-fan-out call sites unless you have > validated them against that admission cap, and make target operations > idempotent because a connection failure can make launch outcome uncertain. See -> the Transaction Semantics section of `USER_GUIDE.md` for details. +> [Transaction Semantics](../USER_GUIDE.md#transaction-semantics) for details. +> Neither mode provides cross-database atomicity with the control store or SQL target. --- @@ -580,8 +596,10 @@ Return columns: encodes sub-orchestration lineage: it starts with the root function instance id and appends a `::{parent_generation}::{branch_or_loop_node_id}` segment for each nested `JOIN`/`RACE` branch and each non-root `df.loop()` (which runs as its own child - sub-orchestration). Instance ids and node ids are 8-char hex and never contain `::`, - so the path is unambiguous. Supersession is evaluated **per scope**: a node is + sub-orchestration). Public instance IDs and node IDs remain 8-char hex. Satellite + execution stamps instead start with the engine root + `pgdf---`; that root contains no `::`, + so existing child composition and path parsing are unchanged. Supersession is evaluated **per scope**: a node is superseded when a newer generation exists for its own `instance_path`, or when any ancestor scope in its path has advanced to a newer generation. For a plain root-level loop this reduces to the second `::`-token being the loop generation. @@ -617,7 +635,7 @@ ORDER BY node_id; ### df.setvar(name, value) -Sets a workflow variable for the current user (before `df.start()`). Each user has their own variable namespace — variables set by one user are invisible to others. +Sets a workflow variable for the current user in this origin database (before `df.start()`). Each user has their own local variable namespace; variables are not shared between installations. `df.setvar` is a setup helper, not a workflow node: do not use it inside `df.seq`, `df.join`, `df.race`, etc. | Parameter | Type | Auto-wrap | Description | @@ -698,15 +716,20 @@ SELECT df.clearvars(); ### df.grant_usage(role_name [, include_http] [, with_grant]) -Grants the privileges a role needs to use pg_durable. By default this grants general `df` usage but does not grant `EXECUTE` on `df.http()`. Pass `include_http => true` to opt a role into HTTP access. Pass `with_grant => true` to allow the role to delegate access to others. +Grants the privileges a role needs in this database's pg_durable installation. +By default, this grants general `df` usage but no HTTP or global metrics access. +`include_http => true` enables `df.http()` and `df.http_multipart()` in this origin. +`with_grant => true` delegates local administration and also grants `df.metrics()`, +exposing aggregate totals across **every origin and user** in the shared engine, +even when the grant is issued in a satellite. Authorization is enforced by PostgreSQL’s native mechanisms: EXECUTE on this function is revoked from PUBLIC (so only roles explicitly granted access can call it), and the inner GRANT statements run as the caller via SECURITY INVOKER, so the caller must hold the underlying privileges WITH GRANT OPTION. | Parameter | Type | Description | |-----------|------|-------------| | `role_name` | TEXT | The role to grant privileges to | -| `include_http` | BOOLEAN | Optional, defaults to `false`; when `true`, also grants `EXECUTE` on `df.http(text, text, text, jsonb, integer)` | -| `with_grant` | BOOLEAN | Optional, defaults to `false`; when `true`, grants all privileges WITH GRANT OPTION and retains EXECUTE on `df.grant_usage` / `df.revoke_usage` | +| `include_http` | BOOLEAN | Optional, defaults to `false`; grants local `EXECUTE` on `df.http()` and `df.http_multipart()` | +| `with_grant` | BOOLEAN | Optional, defaults to `false`; grants local privileges WITH GRANT OPTION, grant/revoke helper access, and shared-engine `df.metrics()` access | ```sql SELECT df.grant_usage('app_role'); @@ -726,6 +749,14 @@ Revokes all privileges previously granted by `df.grant_usage()`, including any ` SELECT df.revoke_usage('app_role'); ``` +### df.metrics() + +Returns shared-engine totals: `total_instances`, `running_instances`, +`completed_instances`, `failed_instances`, `total_executions`, and `total_events`. +These are not filtered by local RLS or origin database. `PUBLIC EXECUTE` is revoked; +access requires a direct administrator grant or +`df.grant_usage(..., with_grant => true)`. Ordinary usage grants omit this function. + --- ## Server Configuration (GUCs) @@ -734,6 +765,55 @@ These settings are configured via `ALTER SYSTEM SET` or `postgresql.conf`. See e --- +### pg_durable.database + +Control database for the single runtime/provider store (default `postgres`, +Postmaster context, restart required). Explicitly install pg_durable here before +satellites. Satellite installs use SQLx with the worker credential to verify control +readiness version `2` or later; extension version strings need not be identical. +This setting is not the default SQL target for satellite starts; their origin is. + +### pg_durable.worker_role + +Connection role for worker management, origin routing, and provider operations +(default `postgres`, a superuser; Postmaster context, restart required). A custom +role needs `CONNECT` and required `df` metadata/guard rights in each origin, plus +access for origin-local HTTP privilege lookup. `BYPASSRLS` grants no database, +schema, table, or function privileges by itself. SQL nodes still authenticate as +the captured submitting role in the execution database. + +### pg_durable.max_origin_connections + +Shared budget for satellite metadata connections across activities and maintenance. + +| Property | Value | +|----------|-------| +| Type | `integer` | +| Default | `12` | +| Range | `2` to `1000` | +| Context | `POSTMASTER` (restart required) | + +Each active route reserves two slots: a guard transaction locking +`df._installation`, `df.instances`, and `df.nodes` in `ACCESS SHARE` mode, and a +metadata connection. Routes close after use; there is no idle pool per database +or database-name count ceiling. Maintenance shares the same budget. The control +pool's `pg_durable.max_management_connections` limit is unchanged and separate. +Origin admission waits up to 30 seconds. Metadata connections use a 1.5-second +lock timeout and a 5-second statement timeout; these are not user SQL timeouts. +Guard transactions disable server idle-in-transaction and transaction timeouts +so long-running activities retain their installation locks. +See [Connection Limits](../USER_GUIDE.md#connection-limits) for total budgeting. + +### pg_durable.max_new_transaction_starts + +Maximum concurrent `transaction_mode => 'new'` loopback launches **per database**, +enforced through advisory locks before connecting. Default `2`, range `1` to +`1000`, Postmaster context (restart required). Launches connect to the caller's +database, not the explicit SQL target. Excess callers wait up to +`pg_durable.new_transaction_start_timeout` seconds (default `5`). + +--- + ### pg_durable.enable_superuser_instances Controls whether pg_durable allows durable function instances whose `submitted_by` role is a PostgreSQL superuser. diff --git a/docs/multi-database-installation.md b/docs/multi-database-installation.md new file mode 100644 index 00000000..caaf151e --- /dev/null +++ b/docs/multi-database-installation.md @@ -0,0 +1,389 @@ +# Multi-Database Extension Installation + +**Status:** Implemented in 0.2.8 +**Date:** 2026-09-10 + +## Summary + +pg_durable already supports executing a workflow's SQL in another database through +the `database` argument to `df.start()`. That feature is documented in +[`multi-database.md`](multi-database.md). This document describes a different +capability: installing `pg_durable` in multiple databases so each database has a +local `df` API, local metadata, local privileges, and local transaction semantics. + +The implemented architecture is: + +- Keep exactly one Duroxide runtime and provider schema in the database selected by + `pg_durable.database`. This is the **control database**. +- Require an explicit control-plane `CREATE EXTENSION pg_durable` in that database. + Preloading starts worker initialization and control-installation checks, but + creates no provider objects before explicit `CREATE EXTENSION`. +- Permit additional **satellite installations** in other databases. Each satellite + owns its local `df` schema, `df.instances`, `df.nodes`, `df.vars`, functions, + privileges, and RLS policies, but no active Duroxide provider schema. +- Namespace satellite engine IDs with the origin database and installation identity. + Activities derive their route from `ActivityContext`; recorded orchestration and + activity payloads are unchanged. Duroxide client operations use the control database. +- Treat the control installation as the lifecycle anchor. Do not attempt automatic + cross-database reference counting for creation or removal of the runtime. + +This preserves the most important local behavior: a normal `df.start()` writes its +graph in the caller's transaction and is rolled back with that transaction. + +## Problem Frame + +PostgreSQL extensions and their dependencies are database-local. The pg_durable +background worker, however, is registered once per cluster from +`shared_preload_libraries`, and the desired Duroxide runtime is also cluster-wide. +Before 0.2.8, the extension could be installed only in `pg_durable.database`. + +Users instead expect this topology: + +```mermaid +flowchart LR + A[Database A
df API and metadata] --> R[One Duroxide runtime] + C[Database C
df API and metadata] --> R + R --> B[Control database B
Duroxide provider schema] + R --> A + R --> C + R --> D[Optional SQL target database D] +``` + +The origin database, Duroxide control database, and SQL execution database are +three distinct concepts. They may be the same database, but the implementation +must not assume that they are. + +## Requirements + +### Installation And Lifecycle + +- **R1.** `CREATE EXTENSION pg_durable` must remain required in + `pg_durable.database` before provider creation and durable execution start. + Worker initialization, management connections, and polling may precede it. +- **R2.** `CREATE EXTENSION pg_durable` must be allowed in additional databases once + a compatible control installation exists. +- **R3.** A satellite installation must create its local `df` API and metadata but + must not create or own a second active Duroxide provider schema. +- **R4.** Dropping a satellite must not stop the shared runtime or remove the + control database's Duroxide schema. +- **R5.** Dropping the control installation must stop the runtime and remove the + provider schema as it does today. Remaining satellites must fail new control-plane + operations with a clear "control installation unavailable" error. +- **R6.** Loading `pg_durable` through `shared_preload_libraries` without a control + installation must continue to leave no Duroxide schema behind. + +### Workflow Behavior + +- **R7.** A workflow started in database A must persist `df.instances`, `df.nodes`, + and `df.vars` state in A, subject to A's RLS and extension privileges. +- **R8.** The default SQL execution database must be the database where + `df.start()` was called. An explicit `database` argument may still target another + database on the cluster. +- **R9.** `transaction_mode => 'caller'` must retain its current commit/rollback + behavior. The worker must probe graph visibility and the originating XID in the + origin database. +- **R10.** Status, result, explain, signal, cancel, await, and instance-listing APIs + called in A must operate on A's local instances while consulting the shared + Duroxide store when engine state is required. +- **R11.** Engine instance identity must be globally unique across installations; + two databases generating the same current eight-character local ID must not + address the same Duroxide orchestration. + +### Security And Operations + +- **R12.** SQL nodes must continue to connect as the captured `current_user` in the + execution database. Installing pg_durable in another database must not grant that + role any new execution privilege there. +- **R13.** The worker credential must be authorized explicitly in every satellite + database it manages. A non-superuser `BYPASSRLS` role still needs the necessary + object privileges and `CONNECT` privilege. +- **R14.** HTTP privilege checks must be evaluated against `df.http()` or + `df.http_multipart()` in the workflow's origin installation, not accidentally + against the control installation. +- **R15.** Connection growth must be bounded independently of the number of + installed databases. Installing 100 satellites must not eagerly allocate 100 + full management pools. +- **R16.** The single worker must tolerate a rolling extension upgrade in which the + control and satellite databases temporarily have different extension schema + versions within the supported binary-compatibility range. + +## Implementation Map + +| Area | Implemented contract | +|---|---| +| [Install DDL](../src/lib.rs) | Local `df` objects everywhere; extension-owned provider namespace only in the control database. Satellite objects belong to the installer; provider objects inside the control namespace are created by `worker_role`. | +| [Origin routing](../src/origin.rs) | Database OID plus installation UUID, short-lived guarded routes, and a shared connection semaphore. | +| [Activity registry](../src/registry.rs) | Route graph admission, metadata updates, SQL defaults, and HTTP authorization using `ActivityContext`. | +| [Backend readiness](../src/types.rs) | Satellites discover the control schema and readiness over SQLx using the worker credential, without a lifetime schema cache. | +| [Worker maintenance](../src/worker.rs) | Retention and orphan reconciliation visit registered origins in bounded batches. | + +## Implemented Architecture + +### 1. Explicit Control Installation + +The control database remains the only owner of the provider schema and the only +database whose extension lifecycle starts or stops the runtime. Administrators use: + +```sql +-- In the database named by pg_durable.database: +CREATE EXTENSION pg_durable; + +-- Then, in each additional database: +CREATE EXTENSION pg_durable; +``` + +Satellite creation verifies over SQLx that the control installation exists and its +worker readiness schema version is at least `2`, which includes the `_origins` +registry. It does not create the control extension automatically. +An automatic remote `CREATE EXTENSION` would commit independently from the local +installation transaction, so a local rollback could leave an unexpected control +installation and provider schema behind. + +This explicit anchor removes the need for lifecycle reference counting. Runtime +initialization follows control creation; shutdown follows detection of control +removal, regardless of satellite count. Removal order is satellites first, control last. + +The provider namespace is extension-owned (`_duroxide` on fresh control installs, +legacy `duroxide` on older upgraded installs). Objects inside it are created by +`pg_durable.worker_role` through worker-only `ApplyAll` migrations and readiness +initialization. Satellite DDL belongs to the local installer and creates neither +provider namespace nor provider objects. + +### 2. Local Metadata, Shared Engine + +Each satellite retains local metadata and local SPI operations. Public instance +IDs remain eight hexadecimal characters. Satellite engine IDs have this form: + +```text +pgdf--- +``` + +The UUID is stored in local `df._installation` and encoded without hyphens in the +engine ID. Database OID permits rename-safe lookup through `pg_database`; the UUID +fences work from a dropped/recreated installation. Activities parse the root ID +from `ActivityContext.instance_id()`, including for existing `::` child-ID suffixes. + +No origin fields are added to recorded orchestration or activity payloads. Child +composition, `continue_as_new`, and control-database replay remain unchanged. +Control instances continue to use their unprefixed local IDs, including on older +schemas without `df._installation`. + +### 3. Database-Aware Activity Routing + +The activity registry uses an origin router for: + +- graph load and transaction admission; +- instance and node status updates; +- HTTP and multipart privilege checks; +- retention and orphan reconciliation. + +SQL execution remains separate. Its effective target is: + +```text +explicit df.start(database => ...) ?? origin database +``` + +For SQL activities, a null execution database defaults to the origin through the +runtime route; an explicit database remains a separate SQL target. +When SQL targets its own satellite, its submitting-user connection revalidates +the database OID and installation UUID after SQL admission. That transaction +retains the installation relation lock through statement execution and commit, +preventing a force-dropped database's queued work from reaching a same-name replacement. + +`pg_durable.max_origin_connections` bounds origin connections across activities and +maintenance: default `12`, minimum `2`, maximum `1000`, Postmaster context (restart +required). Each active route reserves two slots: one installation-guard transaction +and one metadata connection. Routes close their pools when finished; there is no +idle pool per database or database-name count ceiling. The control pool remains +governed by `pg_durable.max_management_connections`; SQL execution connections have +their separate existing budget. + +Origin metadata connections enforce a 1.5-second lock timeout and a 5-second +statement timeout, including secondary activity queries. Guard transactions disable +idle-in-transaction and, where supported, transaction timeouts so a long SQL or +HTTP operation does not silently lose its installation lock. These metadata +deadlines do not set a timeout on user SQL or HTTP requests. + +### 4. Control-Plane Backend Calls + +The following operations use a SQLx/Duroxide client connection to +the control database: + +- start orchestration; +- cancel orchestration; +- raise signal/event; +- fetch instance/execution details; +- fetch system metrics; +- direct calls to Duroxide's published `get_instance_info` function. + +Satellite readiness and provider-schema discovery also use the control connection. +Each satellite backend reuses one direct control-state connection on its cached +runtime, but rechecks extension identity, schema, and readiness on every operation. +Remote probes have 1.5-second server deadlines and a 5-second overall deadline; +failures discard the connection without falling back to stale readiness or a +satellite provider. Control-local callers use SPI so readiness changes in their +own transaction remain visible and do not require an extra control-state connection. +Instance operations authorize through local SPI/RLS before addressing engine state +under the worker credential. `df.metrics()` is the explicit administrative exception: +it reports totals for the entire shared engine, including every satellite. + +`transaction_mode => 'new'` opens its loopback launch session in the caller's +database, regardless of the SQL execution target. Its advisory admission limit, +`pg_durable.max_new_transaction_starts`, is **per database**, not cluster-wide. +Caller-mode graph persistence and transaction admission remain origin-local. + +### 5. Origin Registry For Maintenance, Not Ownership + +The control provider schema's `_origins` table records database OID/installation UUID +pairs idempotently through activity routing for submitted work. `df.start()` does +not synchronously register the origin. Only origins submitting work are discovered, +not all installed extensions. The registry supports maintenance, not ownership or +an exact transactional reference count. + +Exact cross-database reference counting is not reliable with ordinary extension +DDL: extension catalogs and dependencies are database-local, and registration over +a second connection commits or rolls back independently. There is no all-database +discovery scan or `ProcessUtility_hook`; correctness uses local installation locks, +UUID fencing, and eventual reconciliation. + +## Lifecycle And Failure Semantics + +### Satellite Drop + +Dropping a satellite is destructive to that installation's work only. It leaves +the control provider and other satellites intact. During an active satellite +activity, a guard transaction holds `ACCESS SHARE` locks on `df._installation`, +`df.instances`, and `df.nodes`, so `DROP EXTENSION` waits for active operations. +This is not a nonterminal-instance drop veto: sleeping or queued work does not +prevent a drop, and there is no DDL hook or reference count. + +Every activity, including one with a cached graph or an explicit remote SQL target, +must validate the installation UUID before executing. Old work cannot execute +against a recreated installation. Bounded reconciliation cancels running roots +after confirming removal; cancellation does not wait for the retention cutoff. +Deletion of terminal engine records remains subject to retention. Neither action +is immediate at DDL commit. Previously committed SQL or external HTTP effects +are not undone. + +### Control Drop + +Dropping the control extension destroys the shared engine state for **every +satellite** and causes the worker to stop the runtime when it detects the drop. +There is no cross-database dependency or registry veto. Remaining satellite metadata +does not restore the lost engine history; recreating control is not recovery of old +work. Remove satellites first and control last. + +### Database Rename Or Drop + +Origin routing resolves the current database name from its OID before connecting. +Reconciliation treats a missing database OID, missing extension-owned installation +table, or replaced installation UUID as removal. An unreachable database, denied +connection, or failed probe is **not** evidence of absence: cleanup is deferred. +Explicit SQL target names remain names and are not rename-tracked by this route. + +### Mixed Versions + +The shared object is cluster-wide, but `pg_extension.extversion` is per database. +An older supported control schema works with the new binary without local +`df._installation`; its IDs and replay path remain unchanged. Satellites require +the new local identity/validator DDL and control worker readiness version `2`. +This readiness protocol, not equal extension version strings, gates installation. + +## Security Considerations + +- PostgreSQL roles and role OIDs are cluster-wide, but database `CONNECT`, schema, + table, and function privileges are database-local. +- The default worker role is the `postgres` superuser. A custom worker role needs + database-local `CONNECT`, access to `df` metadata and the installation guard, and + permission to perform origin-local HTTP privilege lookup in every managed origin. + `BYPASSRLS` bypasses row policies only; it grants no database or object privileges. +- The caller must never be allowed to supply an arbitrary origin database or + installation ID in raw workflow JSON. The C entrypoint derives origin identity + from the current database and local installation row. +- Engine operations run under the worker credential. Every signal, cancellation, + result lookup, and detailed monitoring operation must first prove local ownership + through SPI/RLS in the origin database. +- An explicit SQL target database still executes as `submitted_by`, so normal + `CONNECT` and SQL privileges remain the execution boundary. +- HTTP authorization is attached to the origin installation. Granting HTTP in + database A must not implicitly grant it in database C. +- `df.grant_usage(..., with_grant => true)` delegates local administration and grants + `df.metrics()` access. Even when granted in a satellite, this exposes all-engine + aggregate totals across users and origins, not merely that satellite's activity. +- Dynamic database and schema identifiers must be resolved from trusted catalog + values and quoted as identifiers; user input remains bound as query parameters. + +## Alternatives Considered + +### Eager Runtime From `shared_preload_libraries` + +Rejected. Creating persistent provider objects on preload would leave them without +an explicit `CREATE EXTENSION` lifecycle owner. Uninstall and downgrade behavior would be +unclear, and removing the library from configuration cannot transactionally clean +database objects. + +### Automatic Cross-Database Reference Counting + +Rejected as the lifecycle foundation. A satellite's extension transaction cannot +atomically update a registry in the control database using an ordinary second +connection. Failed installs, forced drops, database removal, and restore can all +leave the count stale. Eventual registration is still useful for maintenance. + +### Centralize All `df` Metadata In The Control Database + +Rejected. Satellite functions could proxy every operation to the control +database, but `transaction_mode => 'caller'` could no longer naturally couple graph +persistence to the caller's local transaction. It would also require securely +forwarding caller identity for RLS and make local monitoring and grants surprising. + +### One Runtime Per Installed Database + +Rejected for this goal. It gives natural local semantics but multiplies Tokio +runtimes, provider pools, listeners, migrations, retention loops, and resource +budgets. It also contradicts the requirement for one Duroxide runtime and schema. + +## Deferred Limitations + +- Registry discovery begins with work submission, not installation. There is no + complete cross-database installation inventory or administrative instance list. +- Satellite drop has no instant cancellation notification; active-operation locks + and UUID fencing provide safety while reconciliation performs eventual cleanup. +- Local metadata, control enqueue, and explicit SQL targets do not share a + cross-database atomic transaction. Caller-mode admission preserves local + commit/rollback behavior, but does not provide distributed transactions. +- There is no per-node database targeting or idle SQL/metadata pool per satellite. + +## Verification + +The E2E coverage includes origin routing (`14_database`), isolation and installation +replacement (`72_multi_database_lifecycle`), maintenance (`73_multi_database_reconcile`), +lock ordering and cancellation cleanup (`74_multi_database_guards`), and forced +database replacement during SQL admission (`75_multi_database_force_drop`). Pure +tests cover identity parsing, bounded control probes, and retention cursor progress. +The release gates remain full unit/E2E suites, formatting, build, Clippy, and upgrade +testing; focused regressions do not replace them. + +## Upgrade And Migration + +This feature changes local extension DDL, engine identity for satellite starts, and +runtime routing. It does not change recorded orchestration or activity payloads. + +- **B1 binary compatibility:** control IDs bypass the installation-identity lookup, + so supported old schemas without `df._installation` still work. Missing + `df.duroxide_schema()` retains the legacy `duroxide` fallback. Control replay and + existing child composition are unchanged by this feature. +- **Upgrade DDL:** the multi-database additions in + [0.2.7 to 0.2.8](../sql/pg_durable--0.2.7--0.2.8.sql) are `df._installation` + (singleton UUID, public read-only access) and `df.validate_installation()`, invoked + by the upgrade. There is no engine-ID mapping column or provider DDL in this SQL. + The separate loop API change in that script is described in + [Upgrade Testing](upgrade-testing.md#028). +- **Runtime schema detection:** satellites resolve control schema/readiness over + SQLx. The worker creates `_origins` before publishing readiness version `2`; + worker-only `ApplyAll` manages provider migrations in control, never satellites. +- **Upgrade order and gates:** deploy/restart the new binary and wait for a ready + control installation before creating satellites. Equal extension schema versions + are not required. Scenario A must compare like-for-like control installs; B1 + covers all supported old control schemas, and B2 checks retained data and work. + Fresh satellites must have local identity and no provider schema. These remain + validation requirements, not assertions that full gates have passed. \ No newline at end of file diff --git a/docs/multi-database.md b/docs/multi-database.md index 5c9bd7bf..40c93d98 100644 --- a/docs/multi-database.md +++ b/docs/multi-database.md @@ -1,27 +1,34 @@ # Multi-Database Support **Status:** Completed -**Date:** 2026-03-06 +**Updated:** 2026-09-10 ## Summary -Allow durable functions to execute SQL in any database on the same PostgreSQL cluster, not just the database where the extension is installed. A single function always runs against one database; cross-database workflows are deferred to a future enhancement. +Durable functions can execute SQL in any database on the same PostgreSQL cluster. +A single invocation selects one SQL execution database. Since 0.2.8, multiple +databases can also have native local installations sharing one control runtime; +see [Multi-Database Extension Installation](multi-database-installation.md). ## Motivation -Today, pg_durable can only execute SQL in the database configured by `pg_durable.database` (the same database the background worker connects to). Users with multiple databases in the same cluster—e.g., multi-tenant setups, or separate `analytics` / `app` databases—cannot use pg_durable to run durable functions against those databases. +Explicit SQL targeting lets users with separate tenant, `analytics`, or `app` +databases execute work without moving data into the control database. Satellite +installation is a separate capability: it keeps each caller's metadata, grants, +variables, and transaction semantics local. -pg_cron solved the same problem: it stores all metadata in one database but can schedule jobs against any database via `cron.schedule_in_database()`. We adopt a similar approach. +The original target-selection API follows pg_cron's optional remote-execution +model; it does not require an extension installation in the SQL target. ## Design Principles -1. **Extension lives in one database.** The `df` and `duroxide` schemas, background worker connection, and all metadata tables (`df.instances`, `df.nodes`) remain in the database specified by `pg_durable.database`. The extension is created (`CREATE EXTENSION pg_durable`) in only that one database. +1. **One engine, multiple local installations.** Install explicitly in the control database (`pg_durable.database`) first and wait for worker readiness, then install satellites. Each origin owns local `df` metadata, RLS, variables, and APIs. Only control has the provider namespace (`_duroxide`, or legacy `duroxide`); there is one runtime/provider. 2. **One database per function invocation.** A single `df.start()` call targets exactly one database. All SQL nodes in that invocation execute against that database. We explicitly do not support functions that span multiple databases in this iteration—it would complicate the DSL and orchestration for limited benefit. Users needing cross-database work can use `dblink` or `postgres_fdw` inside their SQL queries, or start separate durable functions per database. 3. **DSL is database-agnostic.** The DSL (`df.sql()`, `~>`, `&`, etc.) has no concept of "database." Database is purely a property of the *instance*, set at `df.start()` time. This keeps the DSL simple and avoids a combinatorial explosion of database-aware operators. -4. **Backwards compatible.** Omitting the database parameter defaults to `pg_durable.database` (today's behavior). No existing queries break. +4. **Origin-local default.** Omitting `database` or passing NULL uses the database where `df.start()` was called. Runtime activity routing supplies this default without changing recorded payloads. Existing control-database starts retain their behavior. ## API Design @@ -31,7 +38,7 @@ pg_cron uses a separate function (`cron.schedule_in_database()`). This has the a ### Chosen Approach: Optional Parameter on `df.start()` -Add an optional `database` parameter to `df.start()`: +The optional `database` parameter on `df.start()` selects the SQL target: ```sql -- Existing signature (unchanged behavior): @@ -46,13 +53,8 @@ SELECT df.start(df.sql('SELECT 1'), 'my-label', 'analytics'); The current signature is: ```sql -df.start(fut text, label text DEFAULT NULL) → text -``` - -The new signature becomes: - -```sql -df.start(fut text, label text DEFAULT NULL, database text DEFAULT NULL) → text +df.start(fut text, label text DEFAULT NULL, database text DEFAULT NULL, + transaction_mode text DEFAULT 'caller') → text ``` **Why this is not a breaking change:** @@ -68,12 +70,22 @@ df.start(fut text, label text DEFAULT NULL, database text DEFAULT NULL) → text ### Querying from Other Databases -Users calling `df.start()` from a *different* database than where the extension is installed need to use `dblink` or `postgres_fdw` to call into the extension database. The extension functions (`df.start`, `df.status`, `df.result`, etc.) only exist in the extension database. +Install a satellite after control is ready to call `df.start()`, `df.status()`, +`df.result()`, signal, cancel, await, and other APIs locally. Use public +eight-character IDs in that origin; engine IDs are privately namespaced by database +OID and installation UUID. A database without an installation has no local `df` +API, even if it is a workflow's explicit SQL target. -**Alternative considered:** Installing stub functions in other databases that proxy via `dblink`. Rejected as over-engineering for now; advanced users can set this up themselves. +`transaction_mode => 'caller'` writes metadata in the caller's transaction. +`'new'` uses a loopback session in the caller's database, not the SQL target; +`max_new_transaction_starts` limits these launches per database through advisory +locks. Neither mode makes local metadata, engine state, and remote SQL atomic. ## Schema Changes +The following columns describe the original, already-shipped SQL-target feature; +they are not new 0.2.8 migration DDL. + ### `df.instances` Table Add a `database` column: @@ -82,7 +94,7 @@ Add a `database` column: ALTER TABLE df.instances ADD COLUMN database TEXT; ``` -- `NULL` means "the extension database" (i.e., the database where `df.instances` itself lives). This is always unambiguous: the table only exists in the extension database, so NULL can only refer to that database. Even if `pg_durable.database` were later changed, the old tables would be gone (extension dropped) or still in the original database. +- `NULL` means the origin database where this local `df.instances` row lives. Changing `pg_durable.database` selects a different control store; it does not migrate existing engine state. - Non-NULL values name a different database on the same cluster. - Populated by `df.start()` from the `database` parameter. @@ -102,6 +114,10 @@ The `Durofut` struct (and by extension `df.sql()`, operators, etc.) does not nee ## Implementation Changes +This list records the original explicit-target implementation. Satellite support +adds routing in [src/origin.rs](../src/origin.rs) and +[src/registry.rs](../src/registry.rs), not new recorded orchestration payloads. + ### 1. `df.start()` — [src/dsl.rs](../src/dsl.rs) - Add `database: default!(Option<&str>, "NULL")` parameter. @@ -129,6 +145,9 @@ The `Durofut` struct (and by extension `df.sql()`, operators, etc.) does not nee - Add `database: Option<&str>` parameter. - Use `database.unwrap_or_else(|| &target_database())` for connection options instead of hard-coding `target_database()`. +For satellite work, the activity registry fills a NULL target with the origin +database before calling the SQL activity; the control fallback remains unchanged. + ### 6. Orchestration — [src/orchestrations/execute_function_graph.rs](../src/orchestrations/execute_function_graph.rs) - When building the `ExecuteSqlInput` JSON, include `node.database`. @@ -140,7 +159,8 @@ The `Durofut` struct (and by extension `df.sql()`, operators, etc.) does not nee ### 8. `execute_http` Activity -- No changes needed. HTTP requests don't target a database. +- HTTP requests have no SQL target, but HTTP and multipart authorization is checked + against the origin installation, never against an explicit SQL target. ## Validation @@ -159,87 +179,45 @@ If the database doesn't exist, raise an error immediately rather than letting th - **Role isolation is preserved.** The background worker connects directly as `submitted_by` (the `current_user` captured at `df.start()` time). The user who calls `df.start()` determines the execution role, not the target database. - **`pg_hba.conf` applies.** The background worker's `submitted_by` connection to a different database is subject to the same `pg_hba.conf` rules as any other connection. If the role can't connect to that database, the activity fails with a clear error. - **No privilege escalation.** Targeting a different database doesn't grant additional privileges. SQL executes with `submitted_by`'s permissions *in that database*. +- **Local worker access.** `pg_durable.worker_role` defaults to the `postgres` superuser. A custom role needs database-local `CONNECT`, `df` metadata/guard rights, and access for origin-local HTTP privilege lookup. `BYPASSRLS` does not grant those privileges. +- **Administrative metrics are global.** Local `df.grant_usage(..., with_grant => true)` grants delegation and `df.metrics()` access, which exposes shared-engine totals across all origins and users, including from a satellite. ## Observability - `df.instances` and `df.nodes` gain a `database` column visible in `SELECT * FROM df.instances`. - Background worker logs already include the SQL being executed; adding the database name to log messages in `execute_sql` would be helpful. -- `df.status()` and `df.result()` work unchanged—they query `df.instances`/`df.nodes` which are always in the extension database. +- `df.status()` and `df.result()` use origin-local authorization/metadata and consult the control engine as needed; instance listings remain local and RLS-scoped. -## Migration +## Upgrade and Migration - Existing rows in `df.instances` and `df.nodes` will have `database = NULL`, which correctly means "the extension database." No data migration needed. - The schema change is additive (`ADD COLUMN ... DEFAULT NULL`), safe for rolling upgrades. +- Those target-selection columns predate 0.2.8. The 0.2.8 multi-database upgrade adds + only local installation identity and validation; no provider DDL is added to + migration SQL. The control worker runs `ApplyAll` and creates `_origins` before + publishing readiness schema version `2`, required by satellite installs over SQLx. +- Supported old control schemas still work without `df._installation` or the + provider-schema helper: control IDs stay unprefixed, the missing helper retains + legacy `duroxide` resolution, and control replay/child composition is unchanged. + See [Upgrade Testing](upgrade-testing.md#028). ## Testing -### Unit Tests - -- Verify `df.start()` accepts the new parameter. -- Verify NULL database defaults to `pg_durable.database`. - -### E2E Tests - -- **Same-database (regression):** Existing tests continue to pass without changes. - -- **Cross-database test** (`NN_multi_database.sql`): - - Must run as **superuser** (add to the superuser list in `test-e2e-local.sh`) because it creates/drops a database. However, the durable function itself should be submitted by `df_e2e_user` (non-privileged) to validate that role isolation works across databases. - - ```sql - -- 1. Setup: create test database and grant access to df_e2e_user - CREATE DATABASE test_multi_db; - GRANT CONNECT ON DATABASE test_multi_db TO df_e2e_user; - - -- 2. Create a table in the target database for df_e2e_user - -- (use dblink since we can't switch databases mid-session) - SELECT dblink_exec( - 'dbname=test_multi_db', - 'CREATE TABLE test_tbl (id INT, value TEXT)' - ); - SELECT dblink_exec( - 'dbname=test_multi_db', - 'GRANT ALL ON test_tbl TO df_e2e_user' - ); - - -- 3. Submit durable function as df_e2e_user targeting test_multi_db - SET SESSION AUTHORIZATION df_e2e_user; - CREATE TEMP TABLE _test_state (instance_id TEXT); - INSERT INTO _test_state SELECT df.start( - df.sql('INSERT INTO test_tbl VALUES (1, ''hello'')'), - database => 'test_multi_db' - ); - RESET SESSION AUTHORIZATION; - - -- 4. Poll until complete (standard pattern) - -- ... - - -- 5. Verify the row exists in test_multi_db - SELECT * FROM dblink( - 'dbname=test_multi_db', - 'SELECT value FROM test_tbl WHERE id = 1' - ) AS t(value TEXT); - -- Assert value = 'hello' - - -- 6. Cleanup - DROP TABLE _test_state; - DROP DATABASE test_multi_db; - ``` - - Key aspects this test validates: - - `df.start()` with `database =>` parameter works - - SQL executes in the target database, not the extension database - - Role isolation: function runs as `df_e2e_user`, not the background worker's superuser - - `submitted_by` can connect to the target database (requires `GRANT CONNECT`) - -- **Invalid database:** Verify `df.start(..., database => 'nonexistent')` raises an immediate error (not a deferred background worker failure). +Known passing focused runs are `14_database` (explicit SQL targeting) and +`72_multi_database_lifecycle` (satellite lifecycle). Full unit, E2E, upgrade, and +other release gates are being conducted separately; no broader pass is claimed. ## Scope Exclusions - **Cross-database functions:** A single function graph spanning multiple databases (e.g., read from `db1`, write to `db2`) is not supported. This would require per-node database targeting, which adds significant DSL and orchestration complexity. Users can achieve this via `dblink`/`postgres_fdw` within SQL queries, or by starting separate durable functions per database. -- **Extension installation in multiple databases:** The extension continues to be installed in exactly one database. Supporting multiple installations would require distributed coordination between background workers. +- **Distributed transactions and instant drop cancellation:** Satellite installs are supported, but there is no cross-database atomicity or DDL hook. Active activities guard local installation/metadata with `ACCESS SHARE` locks; a UUID fence blocks stale work after recreation. Drop cleanup is eventual and limited to confirmed removed origins, not unreachable databases. Control drop destroys all satellites' shared engine history. - **Connection pooling per database:** Each SQL activity creates a fresh connection (existing behavior). Per-database connection pooling could improve performance but is orthogonal to this feature. +Origin metadata routes likewise retain no idle pool per database. Activities and +maintenance share `pg_durable.max_origin_connections` (default `12`, range +`2` to `1000`, restart required), reserving two slots per active route. The control +management pool limit is unchanged; there is no database-name count ceiling. + ## Summary of Changes | File | Change | diff --git a/docs/upgrade-testing.md b/docs/upgrade-testing.md index 116d5c90..23fd7d23 100644 --- a/docs/upgrade-testing.md +++ b/docs/upgrade-testing.md @@ -205,6 +205,47 @@ what the upgrade script handles, and any backward compatibility considerations. ### 0.2.8 +#### Multi-database installation + +- **Local DDL:** the multi-database portion of + [0.2.7 to 0.2.8](../sql/pg_durable--0.2.7--0.2.8.sql) adds only + `df._installation` (singleton generated UUID, public read-only access) and + `df.validate_installation()`, then invokes the validator. There is no provider + DDL or engine-ID mapping column in the upgrade SQL. The separate loop changes + below remain in the same script. +- **Control-first installation:** explicitly create the control extension in + `pg_durable.database` and wait for the new worker before creating satellites. + Satellite install/upgrade validates a compatible ready control over SQLx using + the worker credential. Control validation returns immediately because its own + install transaction is not yet visible to the worker. +- **Worker-owned initialization:** readiness schema version `2` includes `_origins`. + The worker creates this registry before publishing readiness; provider migrations + remain worker-only `ApplyAll` in the extension-owned control namespace. Satellite + DDL creates local `df` objects owned by its installer, never `_duroxide` objects. +- **B1:** the new binary retains unprefixed control IDs and bypasses local + `df._installation` lookup in control, so supported old control schemas without + the table still work. Missing `df.duroxide_schema()` retains the legacy + `duroxide` fallback. A new satellite can use an older supported control extension + schema once the new worker publishes readiness `2`; equal extension versions + are not required. +- **Replay/B2:** satellite engine IDs use + `pgdf---`; public IDs remain eight + characters. Activities derive origin from `ActivityContext`, not added recorded + payload fields. Control histories, child composition, and `continue_as_new` + are unchanged by this feature. Active upgrade adds identity/validation without + rewriting existing IDs or histories. Drop/recreate is destructive, not an upgrade. +- **Validation requirements:** Scenario A must compare control fresh/upgrade + schemas like-for-like. Verify satellite-local identity, grants and absence of a + provider schema separately. B1 must cover all supported old control schemas; + B2 must preserve existing data/work. Only the focused `14_database` and + `72_multi_database_lifecycle` runs are currently known to pass; full gates are + being conducted separately, not certified by this documentation update. + +See [Upgrade and Migration](multi-database-installation.md#upgrade-and-migration) +for lifecycle and compatibility boundaries. + +#### Loop API and lifetime + - `sql/pg_durable--0.2.7--0.2.8.sql` renames `df.loop(text, text)` to `df._loop_legacy(text, text)`, preserving its function OID and dependent objects, then creates the single public diff --git a/scripts/test-e2e-local.sh b/scripts/test-e2e-local.sh index e90d83ed..972652be 100755 --- a/scripts/test-e2e-local.sh +++ b/scripts/test-e2e-local.sh @@ -59,6 +59,7 @@ DEFAULT_BUILD_PHASES=( "host-guc" "superuser-guc-off" "connlimit-backpressure" + "force-drop" "connlimit-timeout" "new-start-limit" "connlimit-startup" @@ -71,6 +72,7 @@ ALL_PHASES=( "host-guc" "superuser-guc-off" "connlimit-backpressure" + "force-drop" "connlimit-timeout" "new-start-limit" "connlimit-startup" @@ -145,6 +147,9 @@ phase_label() { connlimit-backpressure) echo "connection limit backpressure" ;; + force-drop) + echo "force-drop execution fence" + ;; connlimit-timeout) echo "connection limit timeout" ;; @@ -183,6 +188,9 @@ phase_for_test() { 44_connection_limit_backpressure) echo "connlimit-backpressure" ;; + 75_multi_database_force_drop) + echo "force-drop" + ;; 45_connection_limit_timeout) echo "connlimit-timeout" ;; @@ -192,7 +200,7 @@ phase_for_test() { 46_connection_limit_startup_validation) echo "connlimit-startup" ;; - 54_reconcile_orphans) + 54_reconcile_orphans|73_multi_database_reconcile) echo "reconcile" ;; 47_http_dsl_disabled) @@ -514,13 +522,15 @@ configure_phase() { set_conf_line "pg_durable.enable_superuser_instances" "on" set_conf_line "pg_durable.max_user_connections" "2" ;; - connlimit-timeout) + force-drop|connlimit-timeout) set_conf_line "shared_preload_libraries" "'pg_durable'" set_conf_line "pg_durable.worker_role" "'postgres'" set_conf_line "pg_durable.database" "'postgres'" set_conf_line "pg_durable.enable_superuser_instances" "on" set_conf_line "pg_durable.max_user_connections" "1" - set_conf_line "pg_durable.execution_acquire_timeout" "2" + if [ "$phase" = "connlimit-timeout" ]; then + set_conf_line "pg_durable.execution_acquire_timeout" "2" + fi ;; new-start-limit) set_conf_line "shared_preload_libraries" "'pg_durable'" @@ -575,7 +585,7 @@ prepare_phase() { http-allow-all) build_extension_http_allow_all ;; - no-preload|standard|host-guc|superuser-guc-off|connlimit-backpressure|connlimit-timeout|connlimit-startup|reconcile) + no-preload|standard|host-guc|superuser-guc-off|connlimit-backpressure|force-drop|connlimit-timeout|connlimit-startup|reconcile) # Rebuild if previous phase changed the Cargo features if [ "$CURRENT_FEATURES" != "http-allow-test-domains" ]; then build_extension @@ -625,7 +635,7 @@ prepare_phase() { wait_for_worker_ready fi ;; - host-guc|connlimit-backpressure|connlimit-timeout) + host-guc|connlimit-backpressure|force-drop|connlimit-timeout) ensure_e2e_role wait_for_worker_ready ;; diff --git a/scripts/test-upgrade.sh b/scripts/test-upgrade.sh index bd04f63f..a2eecbb5 100755 --- a/scripts/test-upgrade.sh +++ b/scripts/test-upgrade.sh @@ -885,6 +885,134 @@ test_b1_instance_info() { assert_sql_equals "SELECT lower(status) FROM df.instance_info('${B1_INSTANCE_ID}');" "completed" } +test_b1_mixed_version_multidb() ( + local control_db="$PG_DB" + local satellite_db="_upgrade_multidb_test" + local satellite_created=false + local control_table_created=false + local control_id satellite_id satellite_engine_id + local sql_client=("$PSQL" -X -h localhost -p "$PG_PORT" -U postgres -qAt -v ON_ERROR_STOP=1) + export PGCONNECT_TIMEOUT=3 + export PGOPTIONS="${PGOPTIONS:+$PGOPTIONS }-c statement_timeout=8000" + + trap ' + status=$? + if [[ "$satellite_created" == true ]]; then + "${sql_client[@]}" -d "$control_db" \ + -c "DROP DATABASE IF EXISTS _upgrade_multidb_test WITH (FORCE);" >/dev/null || status=1 + fi + if [[ "$control_table_created" == true ]]; then + "${sql_client[@]}" -d "$control_db" \ + -c "DROP TABLE IF EXISTS public._upgrade_multidb_log;" >/dev/null || status=1 + fi + exit "$status" + ' EXIT + + assert_sql_equals "SELECT current_database() = df.target_database() + AND (SELECT extversion FROM pg_extension WHERE extname = 'pg_durable') = '${B1_VERSION}' + AND NOT EXISTS (SELECT 1 FROM pg_database WHERE datname = '_upgrade_multidb_test');" "t" || return 1 + case "$B1_VERSION" in + 0.2.[2-7]) + assert_sql_equals "SELECT to_regclass('df._installation') IS NULL;" "t" || return 1 + ;; + esac + if [[ "$B1_VERSION" == "0.2.2" ]]; then + assert_sql_equals "SELECT to_regprocedure('df.duroxide_schema()') IS NULL;" "t" || return 1 + else + assert_sql_equals "SELECT df.duroxide_schema();" "duroxide" || return 1 + fi + assert_sql_equals "SELECT to_regnamespace('_duroxide') IS NULL + AND EXISTS (SELECT 1 FROM duroxide._worker_ready WHERE schema_version >= 2);" "t" || return 1 + + run_sql_capture "DO \$\$ BEGIN + IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'df_e2e_user') THEN + CREATE ROLE df_e2e_user LOGIN; + END IF; + IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'df_e2e_user' + AND rolcanlogin AND NOT rolsuper AND NOT rolbypassrls) THEN + RAISE EXCEPTION 'df_e2e_user must be a non-superuser login without BYPASSRLS'; + END IF; + END \$\$;" >/dev/null || return 1 + run_sql_capture "CREATE TABLE public._upgrade_multidb_log ( + marker TEXT PRIMARY KEY, database_name TEXT NOT NULL, role_name TEXT NOT NULL);" >/dev/null || return 1 + control_table_created=true + run_sql_capture "SELECT df.grant_usage('df_e2e_user'); + GRANT SELECT, INSERT ON public._upgrade_multidb_log TO df_e2e_user;" >/dev/null || return 1 + run_sql_capture "CREATE DATABASE _upgrade_multidb_test TEMPLATE template0;" >/dev/null || return 1 + satellite_created=true + PG_DB="$satellite_db" run_sql_capture "CREATE EXTENSION pg_durable VERSION '${CURRENT_VERSION}'; + SELECT df.grant_usage('df_e2e_user'); + GRANT CONNECT ON DATABASE _upgrade_multidb_test TO df_e2e_user; + CREATE TABLE public._upgrade_multidb_log ( + marker TEXT PRIMARY KEY, database_name TEXT NOT NULL, role_name TEXT NOT NULL); + GRANT SELECT, INSERT ON public._upgrade_multidb_log TO df_e2e_user;" >/dev/null || return 1 + PG_DB="$satellite_db" assert_sql_equals "SELECT + (SELECT extversion FROM pg_extension WHERE extname = 'pg_durable') = '${CURRENT_VERSION}' + AND (SELECT count(*) FROM df._installation WHERE singleton) = 1 + AND df.duroxide_schema() = '_duroxide' + AND to_regnamespace('duroxide') IS NULL AND to_regnamespace('_duroxide') IS NULL;" "t" || return 1 + + control_id=$("${sql_client[@]}" -d "$control_db" -c "SET SESSION AUTHORIZATION df_e2e_user;" \ + -c "SELECT df.start(df.wait_for_signal('b1-multidb-release') ~> 'INSERT INTO public._upgrade_multidb_log + SELECT ''control'', current_database()::text, current_user::text RETURNING database_name', + 'b1-multidb-control');") || return 1 + [[ "$control_id" =~ ^[0-9a-f]{8}$ ]] || return 1 + run_sql_capture "DO \$\$ BEGIN + FOR attempt IN 1..200 LOOP + IF EXISTS (SELECT 1 FROM duroxide.history h JOIN duroxide.instances i + ON i.instance_id = h.instance_id AND i.current_execution_id = h.execution_id + WHERE h.instance_id = '${control_id}' AND h.event_data::jsonb->>'type' = 'ExternalSubscribed' + AND h.event_data::jsonb->>'name' = 'b1-multidb-release') + AND EXISTS (SELECT 1 FROM duroxide.get_instance_info('${control_id}') WHERE status = 'Running') THEN + RETURN; + END IF; + PERFORM pg_sleep(0.01); + END LOOP; + RAISE EXCEPTION 'Legacy control root did not enter its signal wait'; + END \$\$;" >/dev/null || return 1 + satellite_id=$("${sql_client[@]}" -d "$satellite_db" -c "SET SESSION AUTHORIZATION df_e2e_user;" \ + -c "SELECT df.start('WITH written AS (INSERT INTO public._upgrade_multidb_log + SELECT ''satellite'', current_database()::text, current_user::text RETURNING database_name) + SELECT database_name FROM written', 'b1-multidb-satellite');") || return 1 + [[ "$satellite_id" =~ ^[0-9a-f]{8}$ ]] || return 1 + + PG_DB="$satellite_db" assert_sql_equals_ignoring_warnings \ + "SELECT df.wait_for_completion('${satellite_id}', 5);" "completed" || return 1 + assert_sql_equals "SELECT df.status('${control_id}') = 'running' + AND EXISTS (SELECT 1 FROM duroxide.get_instance_info('${control_id}') WHERE status = 'Running');" "t" || return 1 + run_sql_capture "SELECT df.signal('${control_id}', 'b1-multidb-release');" >/dev/null || return 1 + assert_sql_equals_ignoring_warnings \ + "SELECT df.wait_for_completion('${control_id}', 5);" "completed" || return 1 + assert_sql_equals "SELECT count(*) = 1 AND bool_and(marker = 'control' + AND database_name = current_database() AND role_name = 'df_e2e_user') + FROM public._upgrade_multidb_log;" "t" || return 1 + PG_DB="$satellite_db" assert_sql_equals "SELECT count(*) = 1 AND bool_and(marker = 'satellite' + AND database_name = current_database() AND role_name = 'df_e2e_user') + FROM public._upgrade_multidb_log;" "t" || return 1 + assert_sql_equals "SELECT EXISTS (SELECT 1 FROM df.instance_info('${control_id}') WHERE status = 'completed') + AND EXISTS (SELECT 1 FROM df.instances WHERE id = '${control_id}' + AND database IS NULL AND submitted_by = 'df_e2e_user'::regrole) + AND NOT EXISTS (SELECT 1 FROM df.instances WHERE label = 'b1-multidb-satellite');" "t" || return 1 + PG_DB="$satellite_db" assert_sql_equals "SELECT + EXISTS (SELECT 1 FROM df.instance_info('${satellite_id}') WHERE status = 'completed') + AND EXISTS (SELECT 1 FROM df.instances WHERE id = '${satellite_id}' + AND database IS NULL AND submitted_by = 'df_e2e_user'::regrole) + AND NOT EXISTS (SELECT 1 FROM df.instances WHERE label = 'b1-multidb-control') + AND to_regnamespace('duroxide') IS NULL AND to_regnamespace('_duroxide') IS NULL;" "t" || return 1 + satellite_engine_id=$(PG_DB="$satellite_db" run_sql_capture "SELECT 'pgdf-' || + (SELECT oid::text FROM pg_database WHERE datname = current_database()) || '-' || + replace(id::text, '-', '') || '-${satellite_id}' FROM df._installation WHERE singleton;") || return 1 + [[ "$satellite_engine_id" =~ ^pgdf-[0-9]+-[0-9a-f]{32}-${satellite_id}$ ]] || return 1 + assert_sql_equals "SELECT EXISTS (SELECT 1 FROM duroxide.get_instance_info('${control_id}') WHERE status = 'Completed') + AND EXISTS (SELECT 1 FROM duroxide.get_instance_info('${satellite_engine_id}') WHERE status = 'Completed') + AND (SELECT extversion FROM pg_extension WHERE extname = 'pg_durable') = '${B1_VERSION}';" "t" || return 1 + case "$B1_VERSION" in + 0.2.[2-7]) + assert_sql_equals "SELECT to_regclass('df._installation') IS NULL;" "t" || return 1 + ;; + esac +) + # Run B1 tests against each previous version's schema if [ ${#ALL_PREV_VERSIONS[@]} -eq 0 ]; then echo "" @@ -919,6 +1047,7 @@ else run_test "B1 [v${B1_VERSION}]: df.status() on nonexistent" test_b1_status_nonexistent run_test "B1 [v${B1_VERSION}]: df.unsetvar()" test_b1_unsetvar run_test "B1 [v${B1_VERSION}]: df.clearvars()" test_b1_clearvars + run_test "B1 [v${B1_VERSION}]: Current satellite + legacy control overlap" test_b1_mixed_version_multidb done fi diff --git a/sql/pg_durable--0.2.7--0.2.8.sql b/sql/pg_durable--0.2.7--0.2.8.sql index e23a62b6..a0d047d1 100644 --- a/sql/pg_durable--0.2.7--0.2.8.sql +++ b/sql/pg_durable--0.2.7--0.2.8.sql @@ -15,3 +15,21 @@ CREATE FUNCTION df."loop"( ) RETURNS TEXT LANGUAGE c AS 'MODULE_PATHNAME', 'loop_with_policy_wrapper'; + +CREATE TABLE df._installation ( + singleton pg_catalog.bool PRIMARY KEY DEFAULT true CHECK (singleton), + id pg_catalog.uuid NOT NULL DEFAULT pg_catalog.gen_random_uuid() +); +INSERT INTO df._installation (singleton) VALUES (true); +REVOKE ALL ON TABLE df._installation FROM PUBLIC; +GRANT SELECT ON TABLE df._installation TO PUBLIC; + +CREATE FUNCTION df.validate_installation() RETURNS bool +STRICT +LANGUAGE c +AS 'MODULE_PATHNAME', 'validate_installation_wrapper'; + +DO $$ +BEGIN + PERFORM df.validate_installation(); +END $$; diff --git a/src/activities/execute_sql.rs b/src/activities/execute_sql.rs index 62b11c9e..47befc8d 100644 --- a/src/activities/execute_sql.rs +++ b/src/activities/execute_sql.rs @@ -188,6 +188,15 @@ pub async fn execute( ctx: ActivityContext, semaphore: Arc, input_json: String, +) -> Result { + execute_in_origin(ctx, semaphore, input_json, None).await +} + +pub(crate) async fn execute_in_origin( + ctx: ActivityContext, + semaphore: Arc, + input_json: String, + origin: Option, ) -> Result { let input: ExecuteSqlInput = serde_json::from_str(&input_json).map_err(|e| format!("Invalid execute_sql input: {e}"))?; @@ -230,11 +239,51 @@ pub async fn execute( let mut conn = connect_as_user(&input.submitted_by, input.database.as_deref()).await?; + if let Some(origin) = origin { + use sqlx::Connection; + + let mut transaction = conn.begin().await.map_err(|error| error.to_string())?; + let valid: bool = tokio::time::timeout( + std::time::Duration::from_secs(5), + sqlx::query_scalar( + "SELECT EXISTS (SELECT 1 FROM df._installation i + JOIN pg_catalog.pg_depend d ON d.objid = 'df._installation'::regclass + AND d.classid = 'pg_catalog.pg_class'::regclass AND d.deptype = 'e' + JOIN pg_catalog.pg_extension e ON e.oid = d.refobjid AND e.extname = 'pg_durable' + WHERE i.id = $1 AND (SELECT oid FROM pg_catalog.pg_database + WHERE datname = pg_catalog.current_database()) = $2::bigint::oid)", + ) + .bind(origin.installation_id) + .bind(i64::from(origin.database_oid)) + .fetch_one(&mut *transaction), + ) + .await + .map_err(|_| "Origin execution fence timed out".to_string())? + .map_err(|error| format!("Origin execution fence unavailable: {error}"))?; + if !valid { + return Err("Origin installation removed or replaced".to_string()); + } + let result = execute_query(&ctx, &mut transaction, &input.query).await?; + transaction + .commit() + .await + .map_err(|error| error.to_string())?; + return Ok(result); + } + + execute_query(&ctx, &mut conn, &input.query).await +} + +async fn execute_query( + ctx: &ActivityContext, + conn: &mut sqlx::PgConnection, + query: &str, +) -> Result { // SECURITY: Dynamic SQL is intentional. The query is authored by the submitting // user via df.sql() and executes under their own role via connect_as_user(). // This is equivalent to the user running SQL directly. // See docs/spec-security-model.md §4 for the full threat model. - match sqlx::query(&input.query).fetch_all(&mut conn).await { + match sqlx::query(query).fetch_all(conn).await { Ok(rows) => { let mut result_rows: Vec = Vec::new(); for row in &rows { diff --git a/src/activities/update_node_status.rs b/src/activities/update_node_status.rs index 7579acd0..76e3d1f8 100644 --- a/src/activities/update_node_status.rs +++ b/src/activities/update_node_status.rs @@ -5,35 +5,20 @@ use duroxide::ActivityContext; use sqlx::{PgPool, Postgres, QueryBuilder, Transaction}; -use std::sync::atomic::{AtomicU8, Ordering}; use std::sync::Arc; /// Activity name for registration and scheduling pub const NAME: &str = "pg_durable::activity::update-node-status"; -/// Process-global cache for whether df.nodes.status_details exists. -/// -/// 0 = unknown, 1 = present, 2 = absent. The column is added by the -/// 0.2.3 → 0.2.4 upgrade; a binary newer than the schema (Scenario B1) must run -/// against an older schema that lacks it. We cache "present" permanently once -/// seen, but re-probe on "unknown"/"absent" so an in-place ALTER EXTENSION -/// UPDATE that adds the column is picked up without a worker restart. -static STATUS_DETAILS_COL: AtomicU8 = AtomicU8::new(0); - async fn status_details_present(pool: &PgPool) -> bool { - if STATUS_DETAILS_COL.load(Ordering::Relaxed) == 1 { - return true; - } - let present = sqlx::query_scalar::<_, bool>( + sqlx::query_scalar::<_, bool>( "SELECT EXISTS (SELECT 1 FROM information_schema.columns \ WHERE table_schema = 'df' AND table_name = 'nodes' \ AND column_name = 'status_details')", ) .fetch_one(pool) .await - .unwrap_or(false); - STATUS_DETAILS_COL.store(if present { 1 } else { 2 }, Ordering::Relaxed); - present + .unwrap_or(false) } fn execution_id_from_details(status_details: Option<&serde_json::Value>) -> Option<&str> { diff --git a/src/client.rs b/src/client.rs index a5eaba48..52add5d3 100644 --- a/src/client.rs +++ b/src/client.rs @@ -11,14 +11,16 @@ use std::cell::RefCell; use std::sync::OnceLock; +use std::time::Duration; use duroxide::Client; use pgrx::prelude::*; +use sqlx::Connection; use tokio::runtime::Runtime; use crate::types::{ - backend_duroxide_schema, connect_as_user_for_new_transaction, new_backend_provider, - postgres_connection_string, + backend_control_connection_options, connect_as_user_for_new_transaction, new_backend_provider, + postgres_connection_string, read_backend_control_state, BackendControlState, }; /// Cached tokio runtime for client operations. @@ -29,45 +31,57 @@ static CLIENT_RUNTIME: OnceLock = OnceLock::new(); // the client to be reset on connection failures (unlike OnceLock which // is permanent). thread_local! { - static DUROXIDE_CLIENT: RefCell> = const { RefCell::new(None) }; + static DUROXIDE_CLIENT: RefCell> = const { RefCell::new(None) }; + static CONTROL_CONNECTION: RefCell> = const { RefCell::new(None) }; } -/// Check whether the background worker has finished initializing the duroxide -/// schema for the current binary's expected schema version. -/// -/// Returns `false` if `._worker_ready` does not exist, has no -/// row, or has a `schema_version` below `WORKER_SCHEMA_VERSION`. This is a fast -/// SPI read called once per session on the first call to any `df.*` function -/// that needs the duroxide client. -fn is_worker_ready() -> bool { - let schema = backend_duroxide_schema(); - - // First check if the readiness table exists via the catalogue. Querying - // the non-existent table directly would raise a PostgreSQL ERROR that - // aborts the current (sub)transaction — even if caught in Rust. - let table_exists = Spi::get_one_with_args::( - "SELECT EXISTS(SELECT 1 FROM pg_catalog.pg_tables \ - WHERE schemaname = $1 AND tablename = '_worker_ready')", - &[schema.into()], - ) - .ok() - .flatten() - .unwrap_or(false); - - if !table_exists { - return false; +const CONTROL_LOOKUP_TIMEOUT: Duration = Duration::from_secs(5); + +pub(crate) fn backend_control_state(database_url: &str) -> Result { + if let Some(state) = crate::types::backend_local_control_state()? { + return Ok(state); } + CONTROL_CONNECTION.with(|cell| { + get_client_runtime().block_on(refresh_backend_control_state( + database_url, + &mut cell.borrow_mut(), + )) + }) +} - Spi::get_one_with_args::( - &format!( - "SELECT EXISTS(SELECT 1 FROM {}._worker_ready WHERE schema_version >= $1)", - schema - ), - &[crate::WORKER_SCHEMA_VERSION.into()], - ) - .ok() - .flatten() - .unwrap_or(false) +async fn refresh_backend_control_state( + database_url: &str, + cached: &mut Option<(String, sqlx::PgConnection)>, +) -> Result { + if cached.as_ref().is_some_and(|(url, _)| url != database_url) { + *cached = None; + } + + let result = tokio::time::timeout(CONTROL_LOOKUP_TIMEOUT, async { + if cached.is_none() { + let options = backend_control_connection_options(database_url)?; + let connection = sqlx::PgConnection::connect_with(&options) + .await + .map_err(|error| format!("pg_durable control installation unavailable: {error}"))?; + *cached = Some((database_url.to_string(), connection)); + } + let (_, connection) = cached + .as_mut() + .ok_or_else(|| "Control connection unexpectedly missing".to_string())?; + read_backend_control_state(connection).await + }) + .await + .unwrap_or_else(|_| { + Err(format!( + "pg_durable control installation unavailable: control lookup timed out after {}ms", + CONTROL_LOOKUP_TIMEOUT.as_millis() + )) + }); + + if result.is_err() { + *cached = None; + } + result } /// Get or create the cached tokio runtime. @@ -88,22 +102,34 @@ fn with_duroxide_client(f: F) -> Result where F: FnOnce(&Client, &Runtime) -> Result, { - let rt = get_client_runtime(); + let pg_conn_str = postgres_connection_string(); + let control = match backend_control_state(&pg_conn_str) { + Ok(control) if control.ready => control, + result => { + DUROXIDE_CLIENT.with(|cell| *cell.borrow_mut() = None); + return Err(result.err().unwrap_or_else(|| { + "pg_durable control background worker not yet initialized - try again in a moment" + .to_string() + })); + } + }; + DUROXIDE_CLIENT.with(|cell| { + let mut cached = cell.borrow_mut(); + if cached + .as_ref() + .is_some_and(|(url, previous, _)| url != &pg_conn_str || previous != &control) + { + *cached = None; + } + }); + let rt = get_client_runtime(); // Try to use existing client let has_client = DUROXIDE_CLIENT.with(|cell| cell.borrow().is_some()); if !has_client { // Need to create a new client - if !is_worker_ready() { - return Err( - "pg_durable background worker not yet initialized — try again in a moment" - .to_string(), - ); - } - - let pg_conn_str = postgres_connection_string(); - let schema = backend_duroxide_schema(); + let schema = control.schema; let client = rt.block_on(async { // Limit backend provider to 1 connection — backends need minimal // duroxide access (start/cancel/signal only). @@ -120,14 +146,14 @@ where })?; DUROXIDE_CLIENT.with(|cell| { - *cell.borrow_mut() = Some(client); + *cell.borrow_mut() = Some((pg_conn_str, control, client)); }); } // Execute the operation with the client let result = DUROXIDE_CLIENT.with(|cell| { let borrow = cell.borrow(); - let client = borrow + let (_, _, client) = borrow .as_ref() .ok_or_else(|| "Client unexpectedly missing".to_string())?; f(client, rt) @@ -209,7 +235,7 @@ pub fn start_durable_function( ); let fn_name = function_name.to_string(); - let inst_id = instance_id.to_string(); + let inst_id = crate::origin::backend_engine_id(instance_id)?; let inp = input.to_string(); with_duroxide_client(|client, rt| { @@ -342,14 +368,12 @@ pub fn start_in_new_transaction( let label = label.map(|s| s.to_string()); let database = database.map(|s| s.to_string()); let user = user.to_string(); + let origin_database = Spi::get_one::("SELECT pg_catalog.current_database()::text") + .map_err(|e| format!("Failed to resolve caller database: {e}"))? + .ok_or_else(|| "Failed to resolve caller database".to_string())?; rt.block_on(async { - // The extension lives in exactly one database (see docs/multi-database.md), - // so `database=None` resolves to that control database — the same one - // holding the `df` tables this backend writes through SPI. The `database` - // argument is forwarded to the inner `df.start()`, which records it as an - // instance property for the worker to execute against. - let mut conn = connect_as_user_for_new_transaction(&user).await?; + let mut conn = connect_as_user_for_new_transaction(&user, &origin_database).await?; let result = start_on_new_session(&mut conn, &fut, &label, &database).await; @@ -373,7 +397,7 @@ pub fn start_in_new_transaction( /// Cancel a durable function. pub fn cancel_durable_function(instance_id: &str, reason: &str) -> Result<(), String> { - let inst_id = instance_id.to_string(); + let inst_id = crate::origin::backend_engine_id(instance_id)?; let rsn = reason.to_string(); with_duroxide_client(|client, rt| { @@ -389,7 +413,7 @@ pub fn cancel_durable_function(instance_id: &str, reason: &str) -> Result<(), St /// Raise an external event (signal) to a running orchestration. pub fn raise_external_event(instance_id: &str, event_name: &str, data: &str) -> Result<(), String> { - let inst_id = instance_id.to_string(); + let inst_id = crate::origin::backend_engine_id(instance_id)?; let evt_name = event_name.to_string(); let evt_data = data.to_string(); @@ -421,7 +445,85 @@ pub fn raise_external_event(instance_id: &str, event_name: &str, data: &str) -> #[cfg(test)] mod tests { - use super::{format_new_transaction_start_error, is_connection_error}; + use super::{ + backend_control_connection_options, format_new_transaction_start_error, + is_connection_error, refresh_backend_control_state, CONTROL_LOOKUP_TIMEOUT, + }; + use sqlx::Connection; + use std::time::Duration; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + use tokio::net::{TcpListener, TcpStream}; + + async fn accept_control_stub(listener: &TcpListener, authenticate: bool) -> TcpStream { + let (mut socket, _) = listener.accept().await.unwrap(); + let length = socket.read_u32().await.unwrap(); + let mut startup = vec![0; length as usize - 4]; + socket.read_exact(&mut startup).await.unwrap(); + let startup = String::from_utf8(startup).unwrap(); + assert!(startup.contains("-c statement_timeout=1500ms -c lock_timeout=1500ms")); + if authenticate { + socket + .write_all(b"R\0\0\0\x08\0\0\0\0Z\0\0\0\x05I") + .await + .unwrap(); + } + socket + } + + fn assert_control_timeout_closes_socket(cached_connection: bool) { + tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap() + .block_on(async { + tokio::time::timeout(CONTROL_LOOKUP_TIMEOUT + Duration::from_secs(2), async { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let url = format!( + "postgres://worker@{}/control?sslmode=disable", + listener.local_addr().unwrap() + ); + let mut cached = None; + let socket = if cached_connection { + let options = backend_control_connection_options(&url).unwrap(); + let (connection, socket) = tokio::join!( + sqlx::PgConnection::connect_with(&options), + accept_control_stub(&listener, true), + ); + cached = Some((url.clone(), connection.unwrap())); + Some(socket) + } else { + None + }; + let (result, received) = + tokio::join!(refresh_backend_control_state(&url, &mut cached), async { + let mut socket = match socket { + Some(socket) => socket, + None => accept_control_stub(&listener, false).await, + }; + let mut received = Vec::new(); + socket.read_to_end(&mut received).await.unwrap(); + received + }); + assert!(result + .unwrap_err() + .contains("control lookup timed out after 5000ms")); + assert!(cached.is_none()); + assert_eq!(!received.is_empty(), cached_connection); + }) + .await + .expect("control lookup must return and close its socket within the total budget"); + }); + } + + #[test] + fn control_lookup_bounds_connection_setup() { + assert_control_timeout_closes_socket(false); + } + + #[test] + fn control_lookup_discards_stalled_cached_connection() { + assert_control_timeout_closes_socket(true); + } #[test] fn detects_connection_refused() { diff --git a/src/explain.rs b/src/explain.rs index 3f828515..502d1f4b 100644 --- a/src/explain.rs +++ b/src/explain.rs @@ -157,11 +157,18 @@ fn explain_instance(instance_id: &str) -> String { /// Get instance info from Duroxide store fn get_duroxide_instance_info(instance_id: &str) -> (String, Option) { - use crate::types::{backend_duroxide_schema, new_backend_provider, postgres_connection_string}; + use crate::types::{ + new_backend_provider, postgres_connection_string, try_backend_duroxide_schema, + }; use duroxide::Client; + let engine_id = + crate::origin::backend_engine_id(instance_id).unwrap_or_else(|e| pgrx::error!("{e}")); let pg_conn_str = postgres_connection_string(); - let schema = backend_duroxide_schema(); + let schema = match try_backend_duroxide_schema() { + Ok(schema) => schema, + Err(_) => return (String::new(), None), + }; let rt = match tokio::runtime::Builder::new_current_thread() .enable_all() @@ -179,7 +186,7 @@ fn get_duroxide_instance_info(instance_id: &str) -> (String, Option) { let client = Client::new(store); - match client.get_instance_info(instance_id).await { + match client.get_instance_info(&engine_id).await { Ok(info) => (info.status, info.output), Err(_) => (String::new(), None), } diff --git a/src/lib.rs b/src/lib.rs index e9125d94..30a7b768 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -23,6 +23,7 @@ pub static DATABASE: GucSetting> = pub static HOST: GucSetting> = GucSetting::>::new(Some(c"")); pub static MAX_MANAGEMENT_CONNECTIONS: GucSetting = GucSetting::::new(6); +pub static MAX_ORIGIN_CONNECTIONS: GucSetting = GucSetting::::new(12); pub static MAX_DUROXIDE_CONNECTIONS: GucSetting = GucSetting::::new(10); pub static MAX_USER_CONNECTIONS: GucSetting = GucSetting::::new(10); pub static MAX_NEW_TRANSACTION_STARTS: GucSetting = GucSetting::::new(2); @@ -70,6 +71,7 @@ pub mod explain; pub mod monitoring; pub mod node_status; pub mod orchestrations; +pub(crate) mod origin; pub mod redact; pub mod registry; pub mod ssrf; @@ -83,7 +85,7 @@ pub use types::Durofut; /// by the background worker after successful initialization. Increment whenever /// a new binary introduces new duroxide-pg migration scripts or any other /// BGW-applied duroxide schema change. -pub const WORKER_SCHEMA_VERSION: i32 = 1; +pub const WORKER_SCHEMA_VERSION: i32 = 2; ::pgrx::pg_module_magic!(name, version); @@ -137,6 +139,17 @@ pub extern "C-unwind" fn _PG_init() { GucFlags::default(), ); + GucRegistry::define_int_guc( + c"pg_durable.max_origin_connections", + c"Maximum total satellite metadata connections across activities and maintenance", + c"Each active satellite route reserves two connections. Requires a server restart.", + &MAX_ORIGIN_CONNECTIONS, + 2, + 1000, + GucContext::Postmaster, + GucFlags::default(), + ); + GucRegistry::define_int_guc( c"pg_durable.max_duroxide_connections", c"Maximum number of connections in the duroxide provider pool (orchestration state + listener)", @@ -161,7 +174,7 @@ pub extern "C-unwind" fn _PG_init() { GucRegistry::define_int_guc( c"pg_durable.max_new_transaction_starts", - c"Maximum number of concurrent transaction_mode => 'new' df.start() loopback launch sessions", + c"Maximum concurrent transaction_mode => 'new' df.start() loopback launch sessions per database", c"", &MAX_NEW_TRANSACTION_STARTS, 1, @@ -256,14 +269,24 @@ pub extern "C-unwind" fn _PG_init() { // Schema Declaration // ============================================================================ -// Create both extension-owned schemas as the very first statements of the +// Create the local schema and the control-only provider schema at the start of the // install script. `bootstrap` guarantees this runs before every other extension // object, including the redundant `CREATE SCHEMA IF NOT EXISTS df` that pgrx // emits for the `#[pg_schema] mod df` entity below. extension_sql!( r#" CREATE SCHEMA df; -CREATE SCHEMA _duroxide; +DO $$ +DECLARE + target_db pg_catalog.text := pg_catalog.current_setting('pg_durable.database', true); +BEGIN + IF target_db IS NULL OR target_db OPERATOR(pg_catalog.=) '' THEN + target_db := 'postgres'; + END IF; + IF pg_catalog.current_database() OPERATOR(pg_catalog.=) target_db THEN + CREATE SCHEMA _duroxide; + END IF; +END $$; -- Returns the name of the duroxide provider schema selected for this install. -- Fresh installs return '_duroxide'. The body is version-specific: the upgrade @@ -391,6 +414,14 @@ CREATE TABLE df._worker_epoch ( last_seen_at TIMESTAMPTZ DEFAULT pg_catalog.now() ); +CREATE TABLE df._installation ( + singleton pg_catalog.bool PRIMARY KEY DEFAULT true CHECK (singleton), + id pg_catalog.uuid NOT NULL DEFAULT pg_catalog.gen_random_uuid() +); +INSERT INTO df._installation (singleton) VALUES (true); +REVOKE ALL ON TABLE df._installation FROM PUBLIC; +GRANT SELECT ON TABLE df._installation TO PUBLIC; + ALTER TABLE df.instances ADD CONSTRAINT instances_id_format_chk -- Operators (OPERATOR(pg_catalog.)) and functions (e.g. pg_catalog.now) @@ -507,7 +538,7 @@ CREATE POLICY vars_user_isolation ON df.vars USING (owner OPERATOR(pg_catalog.=) pg_catalog.quote_ident(current_user)::pg_catalog.regrole) WITH CHECK (owner OPERATOR(pg_catalog.=) pg_catalog.quote_ident(current_user)::pg_catalog.regrole); --- No automatic PUBLIC grants — admins call df.grant_usage('role') after +-- No automatic PUBLIC schema access — admins call df.grant_usage('role') after -- CREATE EXTENSION (or see USER_GUIDE.md "Privilege Grants" for manual GRANTs). -- Helper: grant all required df privileges to a role in one call. Additive @@ -675,40 +706,23 @@ REVOKE EXECUTE ON FUNCTION df.revoke_usage(text) FROM PUBLIC; ); // ============================================================================ -// Extension Validation (must run before duroxide schema creation) +// Extension Validation // ============================================================================ -// In production builds, validate that the extension is created in the database -// the background worker will connect to. In pgrx test builds the test database -// name is chosen by pgrx and won't match the worker's target database, so we -// skip the check (unit tests don't need the background worker). +// Satellite installs require a ready, compatible control installation. Control +// installs return immediately because their DDL is not visible to the worker yet. +// pgrx unit tests do not require a running control installation. #[cfg(not(any(test, feature = "pg_test")))] extension_sql!( r#" --- Validate that CREATE EXTENSION is run in the correct database --- The background worker connects to one specific database (determined by --- the pg_durable.database GUC, defaults to "postgres"). --- The extension must be created in that database for workflows to execute. DO $$ -DECLARE - current_db TEXT; - target_db TEXT; BEGIN - -- Get the current database - SELECT pg_catalog.current_database() INTO current_db; - - -- Get the target database that the background worker will connect to - SELECT df.target_database() INTO target_db; - - IF current_db OPERATOR(pg_catalog.<>) target_db THEN - RAISE EXCEPTION 'pg_durable extension must be created in database "%" (currently in "%"). The background worker only processes functions in the database specified by the pg_durable.database GUC (defaults to "postgres").', target_db, current_db - USING HINT = 'Connect to the correct database and run: CREATE EXTENSION pg_durable;'; - END IF; + PERFORM df.validate_installation(); END $$; "#, name = "validate_database", - requires = [df, target_database] + requires = [df, origin::validate_installation, "create_tables"] ); #[cfg(any(test, feature = "pg_test"))] @@ -1664,13 +1678,20 @@ mod tests { #[pg_test] fn test_connection_info_builders() { - use crate::types::{backend_duroxide_schema, postgres_connection_string}; + use crate::types::{backend_provider_config, postgres_connection_string}; let conn = postgres_connection_string(); assert!(!conn.is_empty()); assert!(conn.contains("postgres://")); // Fresh installs use the "_duroxide" provider schema; upgraded installs // use the legacy "duroxide". Both contain "duroxide" as a substring. - assert!(backend_duroxide_schema().contains("duroxide")); + for schema in ["_duroxide", "duroxide"] { + assert_eq!( + backend_provider_config(&conn, schema) + .schema_name + .as_deref(), + Some(schema) + ); + } } #[pg_test] diff --git a/src/monitoring.rs b/src/monitoring.rs index 6b2aeef5..9af38bee 100644 --- a/src/monitoring.rs +++ b/src/monitoring.rs @@ -154,6 +154,11 @@ fn fetch_instance_info_map( pg_conn_str: &str, provider_schema: &str, ) -> HashMap)> { + let engine_ids: Vec = ids + .iter() + .map(|id| crate::origin::backend_engine_id(id)) + .collect::>() + .unwrap_or_else(|e| pgrx::error!("{e}")); let rt = match tokio::runtime::Builder::new_current_thread() .enable_all() .build() @@ -165,8 +170,6 @@ fn fetch_instance_info_map( rt.block_on(async { use sqlx::postgres::PgPoolOptions; - let mut info_by_id: HashMap)> = HashMap::new(); - let monitoring_conn_str = connection_url_with_application_name(pg_conn_str, BACKEND_MONITORING_APPLICATION_NAME); let pool = match PgPoolOptions::new() @@ -182,7 +185,7 @@ fn fetch_instance_info_map( // looking like "no instances". Err(e) => { pgrx::warning!("df.list_instances: could not connect to duroxide store: {e}"); - return info_by_id; + return HashMap::new(); } }; @@ -194,7 +197,7 @@ fn fetch_instance_info_map( ); let rows = match sqlx::query_as::<_, (String, String, i64, Option)>(&batch_sql) - .bind(ids) + .bind(&engine_ids) .fetch_all(&pool) .await { @@ -209,14 +212,74 @@ fn fetch_instance_info_map( } }; - for (id, function_name, execution_count, output) in rows { - info_by_id.insert(id, (function_name, execution_count, output)); - } - - info_by_id + pool.close().await; + local_instance_info_map(ids, &engine_ids, rows) }) } +fn local_instance_info_map( + local_ids: &[String], + engine_ids: &[String], + rows: Vec<(String, String, i64, Option)>, +) -> HashMap)> { + let local_by_engine: HashMap<_, _> = engine_ids.iter().zip(local_ids).collect(); + rows.into_iter() + .filter_map(|(engine_id, function_name, execution_count, output)| { + let local_id = local_by_engine.get(&engine_id)?; + Some(( + (*local_id).clone(), + (function_name, execution_count, output), + )) + }) + .collect() +} + +#[cfg(test)] +mod instance_info_map_tests { + use super::local_instance_info_map; + + #[test] + fn maps_namespaced_rows_to_local_keys_without_relying_on_row_order() { + let local_ids = vec!["deadbeef".into(), "cafebabe".into(), "12345678".into()]; + let engine_ids = vec![ + "origin-a/deadbeef".into(), + "origin-a/cafebabe".into(), + "origin-a/12345678".into(), + ]; + let rows = vec![ + ("origin-a/cafebabe".into(), "second".into(), 2, None), + ("origin-b/deadbeef".into(), "foreign".into(), 9, None), + ( + "origin-a/deadbeef".into(), + "first".into(), + 1, + Some("result".into()), + ), + ]; + + let mapped = local_instance_info_map(&local_ids, &engine_ids, rows); + + assert_eq!(mapped.len(), 2); + assert_eq!( + mapped["deadbeef"], + ("first".into(), 1, Some("result".into())) + ); + assert_eq!(mapped["cafebabe"], ("second".into(), 2, None)); + assert!(!mapped.contains_key("12345678")); + } + + #[test] + fn preserves_legacy_control_ids() { + let ids = vec!["deadbeef".into()]; + let rows = vec![("deadbeef".into(), "legacy".into(), 3, None)]; + + let mapped = local_instance_info_map(&ids, &ids, rows); + + assert_eq!(mapped.len(), 1); + assert_eq!(mapped["deadbeef"], ("legacy".into(), 3, None)); + } +} + /// List durable function instances, newest-first, optionally filtered by status. /// /// This is the original two-argument monitoring entry point. The richer @@ -244,9 +307,6 @@ pub fn list_instances( > { enforce_list_instances_limit(limit_count); - let pg_conn_str = postgres_connection_string(); - let provider_schema = backend_duroxide_schema(); - // Query df.instances via SPI first — RLS filters to calling user's rows only. // We also fetch status here so that all three monitoring APIs (df.status(), // df.list_instances(), df.instance_info()) share the same authoritative source @@ -284,6 +344,8 @@ pub fn list_instances( } let ids: Vec = user_instances.iter().map(|(id, _, _)| id.clone()).collect(); + let pg_conn_str = postgres_connection_string(); + let provider_schema = backend_duroxide_schema(); let mut info_by_id = fetch_instance_info_map(&ids, &pg_conn_str, provider_schema); // Reassemble in df.instances order (created_at DESC). Instances with no @@ -374,9 +436,6 @@ pub fn list_instances_paged( None => None, }; - let pg_conn_str = postgres_connection_string(); - let provider_schema = backend_duroxide_schema(); - // Query df.instances via SPI first — RLS filters to calling user's rows only. // We fetch status, created_at and completed_at here so that all three // monitoring APIs (df.status(), df.list_instances(), df.instance_info()) share @@ -507,6 +566,8 @@ pub fn list_instances_paged( // fetch_instance_info_map). The id set is the already RLS-filtered ids above, // and status is taken from df.instances so all monitoring APIs agree on it. let ids: Vec = user_instances.iter().map(|(id, ..)| id.clone()).collect(); + let pg_conn_str = postgres_connection_string(); + let provider_schema = backend_duroxide_schema(); let mut info_by_id = fetch_instance_info_map(&ids, &pg_conn_str, provider_schema); // Reassemble in df.instances order (created_at DESC, id ASC). Instances with @@ -569,8 +630,6 @@ pub fn instance_info( name!(output, Option), ), > { - let pg_conn_str = postgres_connection_string(); - let provider_schema = backend_duroxide_schema(); let instance_id_str = instance_id.to_string(); // Ownership check: SPI goes through RLS, returning NULL for non-owned instances. @@ -599,6 +658,10 @@ pub fn instance_info( None => return TableIterator::new(vec![]), }; + let engine_id = + crate::origin::backend_engine_id(instance_id).unwrap_or_else(|e| pgrx::error!("{e}")); + let pg_conn_str = postgres_connection_string(); + let provider_schema = backend_duroxide_schema(); let rt = match tokio::runtime::Builder::new_current_thread() .enable_all() .build() @@ -615,9 +678,9 @@ pub fn instance_info( let client = Client::new(store); - match client.get_instance_info(&instance_id_str).await { + match client.get_instance_info(&engine_id).await { Ok(info) => vec![( - info.instance_id, + instance_id_str, label, info.orchestration_name, info.orchestration_version, @@ -665,10 +728,6 @@ pub fn instance_executions( // (PR5 / #146); give it its own bound if per-instance history ever needs one. let limit_count = limit_count.min(10000); - let pg_conn_str = postgres_connection_string(); - let provider_schema = backend_duroxide_schema(); - let instance_id_owned = instance_id.to_string(); - // Ownership check: SPI goes through RLS, so non-owned instances are invisible. // A non-existent or non-owned instance legitimately has no history to show, // so an empty rowset (not an error) is the correct response here. @@ -684,6 +743,10 @@ pub fn instance_executions( return TableIterator::new(vec![]); } + let engine_id = + crate::origin::backend_engine_id(instance_id).unwrap_or_else(|e| pgrx::error!("{e}")); + let pg_conn_str = postgres_connection_string(); + let provider_schema = backend_duroxide_schema(); let rt = match tokio::runtime::Builder::new_current_thread() .enable_all() .build() @@ -699,7 +762,7 @@ pub fn instance_executions( let client = Client::new(store); let execution_ids = client - .list_executions(&instance_id_owned) + .list_executions(&engine_id) .await .map_err(|e| format!("failed to list executions: {e:?}"))?; @@ -710,7 +773,7 @@ pub fn instance_executions( let mut rows = Vec::new(); for exec_id in limited { let info = client - .get_execution_info(&instance_id_owned, exec_id) + .get_execution_info(&engine_id, exec_id) .await .map_err(|e| format!("failed to fetch info for execution {exec_id}: {e:?}"))?; @@ -741,7 +804,8 @@ pub fn instance_executions( /// Access is controlled by PostgreSQL function privileges. Roles with ordinary /// df usage can call `df.list_instances()` to see counts scoped to their own /// workflows; `df.metrics()` should be granted only to roles that may see -/// system-wide aggregate counts. +/// system-wide aggregate counts. A database-local grant in any satellite exposes +/// counts across every installation sharing the control database, not local counts. #[pg_extern(schema = "df")] pub fn metrics() -> TableIterator< 'static, diff --git a/src/origin.rs b/src/origin.rs new file mode 100644 index 00000000..886fe2ec --- /dev/null +++ b/src/origin.rs @@ -0,0 +1,277 @@ +use pgrx::prelude::*; +use sqlx::{postgres::PgConnectOptions, PgPool, Postgres, Transaction}; +use std::{ + str::FromStr, + sync::{Arc, OnceLock}, + time::Duration, +}; +use tokio::sync::{OwnedSemaphorePermit, Semaphore}; +use uuid::Uuid; + +use crate::types; + +static ORIGIN_CONNECTION_SLOTS: OnceLock> = OnceLock::new(); + +pub(crate) async fn acquire_connections(count: u32) -> Result { + let slots = ORIGIN_CONNECTION_SLOTS + .get_or_init(|| Arc::new(Semaphore::new(crate::MAX_ORIGIN_CONNECTIONS.get() as usize))); + tokio::time::timeout( + Duration::from_secs(30), + slots.clone().acquire_many_owned(count), + ) + .await + .map_err(|_| "Origin connection admission timed out")? + .map_err(|_| "Origin connection admission closed".to_string()) +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct Origin { + pub database_oid: u32, + pub installation_id: Uuid, +} + +impl Origin { + pub fn engine_id(&self, local_id: &str) -> String { + format!( + "pgdf-{}-{}-{local_id}", + self.database_oid, + self.installation_id.simple() + ) + } + + pub fn from_engine_id(engine_id: &str) -> Result, String> { + let root = engine_id.split("::").next().unwrap_or(engine_id); + let Some(encoded) = root.strip_prefix("pgdf-") else { + return Ok(None); + }; + let fields: Vec<_> = encoded.split('-').collect(); + if fields.len() != 3 + || fields[2].len() != 8 + || !fields[2].bytes().all(|byte| byte.is_ascii_hexdigit()) + { + return Err("Invalid pg_durable engine instance identity".to_string()); + } + let database_oid = fields[0] + .parse::() + .map_err(|_| "Invalid origin database OID".to_string())?; + let installation_id = Uuid::parse_str(fields[1]) + .map_err(|_| "Invalid origin installation identity".to_string())?; + Ok(Some(Self { + database_oid, + installation_id, + })) + } +} + +pub(crate) fn backend_engine_id(local_id: &str) -> Result { + let database = Spi::get_one::("SELECT pg_catalog.current_database()::text") + .map_err(|error| error.to_string())? + .ok_or("Caller database is unavailable")?; + if database == types::get_database() { + return Ok(local_id.to_string()); + } + let installation_id = Spi::get_one::("SELECT id::text FROM df._installation") + .map_err(|error| format!("Origin installation unavailable: {error}"))? + .ok_or("Origin installation identity is missing")?; + let origin = Origin { + database_oid: unsafe { pgrx::pg_sys::MyDatabaseId.to_u32() }, + installation_id: Uuid::parse_str(&installation_id).map_err(|error| error.to_string())?, + }; + Ok(origin.engine_id(local_id)) +} + +#[pg_extern(schema = "df")] +pub fn validate_installation() -> bool { + let database = Spi::get_one::("SELECT pg_catalog.current_database()::text") + .unwrap_or_else(|error| pgrx::error!("Cannot identify installation database: {error}")); + if database.as_deref() == Some(types::get_database().as_str()) { + return true; + } + let state = types::backend_control_state(&types::postgres_connection_string()) + .unwrap_or_else(|error| pgrx::error!("{error}")); + if !state.ready { + pgrx::error!("pg_durable control installation unavailable or not ready"); + } + true +} + +pub(crate) struct Router { + control: Arc, + connection_options: PgConnectOptions, +} + +pub(crate) struct Route { + pub pool: Arc, + pub database: Option, + guard: Option>, + permit: Option, +} + +impl Route { + pub async fn close(mut self) { + if let Some(guard) = self.guard.take() { + let _ = guard.rollback().await; + } + if self.permit.is_some() { + self.pool.close().await; + } + } +} + +impl Drop for Route { + fn drop(&mut self) { + if let Some(permit) = self.permit.take() { + let pool = self.pool.clone(); + let guard = self.guard.take(); + tokio::spawn(async move { + if let Some(guard) = guard { + let _ = guard.rollback().await; + } + pool.close().await; + drop(permit); + }); + } + } +} + +impl Router { + pub fn new(control: Arc) -> Self { + Self { + control, + connection_options: PgConnectOptions::from_str(&types::postgres_connection_string()) + .expect("valid worker connection configuration") + .application_name(types::WORKER_MANAGEMENT_APPLICATION_NAME), + } + } + + pub async fn route(&self, engine_id: &str) -> Result { + let Some(origin) = Origin::from_engine_id(engine_id)? else { + return Ok(Route { + pool: self.control.clone(), + database: None, + guard: None, + permit: None, + }); + }; + crate::worker::register_origin(&self.control, &origin).await?; + self.connect(&origin).await + } + + pub async fn connect(&self, origin: &Origin) -> Result { + let permit = acquire_connections(2).await?; + let database: String = sqlx::query_scalar( + "SELECT datname FROM pg_catalog.pg_database WHERE oid = $1::bigint::oid AND datallowconn", + ) + .bind(i64::from(origin.database_oid)) + .fetch_optional(self.control.as_ref()) + .await + .map_err(|error| format!("Origin database lookup failed: {error}"))? + .ok_or("Origin database removed or connections disabled")?; + let pool = sqlx::postgres::PgPoolOptions::new() + .max_connections(2) + .acquire_timeout(Duration::from_secs(5)) + .after_connect(|connection, _| { + Box::pin(async move { + sqlx::query( + "SELECT pg_catalog.set_config('transaction_timeout', '0', false) + WHERE pg_catalog.current_setting('transaction_timeout', true) IS NOT NULL", + ) + .execute(connection) + .await?; + Ok(()) + }) + }) + .connect_with( + self.connection_options + .clone() + .database(&database) + .options([ + ("lock_timeout", "1500ms"), + ("statement_timeout", "5s"), + ("idle_in_transaction_session_timeout", "0"), + ]), + ) + .await + .map_err(|error| format!("Origin database connection failed: {error}"))?; + let mut route = Route { + pool: Arc::new(pool), + database: Some(database), + guard: None, + permit: Some(permit), + }; + let mut guard = route + .pool + .begin() + .await + .map_err(|error| error.to_string())?; + sqlx::query("LOCK TABLE df._installation, df.instances, df.nodes IN ACCESS SHARE MODE") + .execute(&mut *guard) + .await + .map_err(|error| error.to_string())?; + let valid: bool = sqlx::query_scalar( + "SELECT EXISTS (SELECT 1 FROM df._installation i + JOIN pg_catalog.pg_class c ON c.oid = 'df._installation'::regclass + JOIN pg_catalog.pg_depend d ON d.objid = c.oid + AND d.classid = 'pg_catalog.pg_class'::regclass AND d.deptype = 'e' + JOIN pg_catalog.pg_extension e ON e.oid = d.refobjid AND e.extname = 'pg_durable' + WHERE i.id = $1 AND (SELECT oid FROM pg_catalog.pg_database + WHERE datname = pg_catalog.current_database()) = $2::bigint::oid)", + ) + .bind(origin.installation_id) + .bind(i64::from(origin.database_oid)) + .fetch_one(&mut *guard) + .await + .map_err(|error| error.to_string())?; + if !valid { + return Err("Origin installation removed or replaced".to_string()); + } + route.guard = Some(guard); + Ok(route) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn multi_database_identity_preserves_child_routing() { + let origin = Origin { + database_oid: 42, + installation_id: Uuid::from_u128(7), + }; + let engine_id = origin.engine_id("deadbeef"); + assert_eq!(Origin::from_engine_id(&engine_id), Ok(Some(origin.clone()))); + assert_eq!( + Origin::from_engine_id(&format!("{engine_id}::2::cafebabe::3::12345678")), + Ok(Some(origin)) + ); + } + + #[test] + fn multi_database_identity_is_scoped_to_installation_and_database() { + let origin = Origin { + database_oid: 42, + installation_id: Uuid::from_u128(7), + }; + let other_database = Origin { + database_oid: 43, + ..origin.clone() + }; + let recreated = Origin { + installation_id: Uuid::from_u128(8), + ..origin.clone() + }; + assert_ne!( + origin.engine_id("deadbeef"), + other_database.engine_id("deadbeef") + ); + assert_ne!( + origin.engine_id("deadbeef"), + recreated.engine_id("deadbeef") + ); + assert_eq!(Origin::from_engine_id("deadbeef"), Ok(None)); + assert_eq!(Origin::from_engine_id("deadbeef::1::cafebabe"), Ok(None)); + assert!(Origin::from_engine_id("pgdf-bad").is_err()); + } +} diff --git a/src/registry.rs b/src/registry.rs index 1eee1517..a3661f68 100644 --- a/src/registry.rs +++ b/src/registry.rs @@ -15,47 +15,149 @@ use crate::orchestrations; /// Create the activity registry with all registered activities pub fn create_activity_registry(pool: Arc, semaphore: Arc) -> ActivityRegistry { let sql_semaphore = semaphore; - let graph_pool = pool.clone(); - let transaction_graph_pool = pool.clone(); - let status_pool = pool.clone(); - let node_status_pool = pool.clone(); - let http_pool = pool.clone(); - let multipart_pool = pool.clone(); + let router = Arc::new(crate::origin::Router::new(pool)); + let sql_pool = router.clone(); + let graph_pool = router.clone(); + let transaction_graph_pool = router.clone(); + let status_pool = router.clone(); + let node_status_pool = router.clone(); + let http_pool = router.clone(); + let multipart_pool = router; ActivityRegistry::builder() - .register(activities::execute_sql::NAME, move |ctx: ActivityContext, input_json: String| { - let sem = sql_semaphore.clone(); - async move { activities::execute_sql::execute(ctx, sem, input_json).await } - }) - .register(activities::load_function_graph::NAME, move |ctx: ActivityContext, instance_id: String| { - let pool = graph_pool.clone(); - async move { activities::load_function_graph::execute(ctx, pool, instance_id).await } - }) + .register( + activities::execute_sql::NAME, + move |ctx: ActivityContext, input_json: String| { + let sem = sql_semaphore.clone(); + let router = sql_pool.clone(); + async move { + let route = router.route(ctx.instance_id()).await?; + let mut execution_origin = None; + let input_json = if let Some(database) = route.database.as_deref() { + let mut input: activities::execute_sql::ExecuteSqlInput = + serde_json::from_str(&input_json).map_err(|error| error.to_string())?; + if input.database.is_none() { + input.database = Some(database.to_string()); + } + if input.database.as_deref() == Some(database) { + execution_origin = + crate::origin::Origin::from_engine_id(ctx.instance_id())?; + } + serde_json::to_string(&input).map_err(|error| error.to_string())? + } else { + input_json + }; + let result = activities::execute_sql::execute_in_origin( + ctx, + sem, + input_json, + execution_origin, + ) + .await; + route.close().await; + result + } + }, + ) + .register( + activities::load_function_graph::NAME, + move |ctx: ActivityContext, instance_id: String| { + let pool = graph_pool.clone(); + async move { + let route = pool.route(ctx.instance_id()).await?; + let result = activities::load_function_graph::execute( + ctx, + route.pool.clone(), + instance_id, + ) + .await; + route.close().await; + result + } + }, + ) .register( activities::load_function_graph::TRANSACTION_AWARE_NAME, move |ctx: ActivityContext, input_json: String| { let pool = transaction_graph_pool.clone(); async move { - activities::load_function_graph::probe_transaction(ctx, pool, input_json).await + let route = pool.route(ctx.instance_id()).await?; + let result = activities::load_function_graph::probe_transaction( + ctx, + route.pool.clone(), + input_json, + ) + .await; + route.close().await; + result + } + }, + ) + .register( + activities::update_instance_status::NAME, + move |ctx: ActivityContext, input_json: String| { + let pool = status_pool.clone(); + async move { + let route = pool.route(ctx.instance_id()).await?; + let result = activities::update_instance_status::execute( + ctx, + route.pool.clone(), + input_json, + ) + .await; + route.close().await; + result + } + }, + ) + .register( + activities::update_node_status::NAME, + move |ctx: ActivityContext, input_json: String| { + let pool = node_status_pool.clone(); + async move { + let route = pool.route(ctx.instance_id()).await?; + let result = activities::update_node_status::execute( + ctx, + route.pool.clone(), + input_json, + ) + .await; + route.close().await; + result + } + }, + ) + .register( + activities::execute_http::NAME, + move |ctx: ActivityContext, config_json: String| { + let pool = http_pool.clone(); + async move { + let route = pool.route(ctx.instance_id()).await?; + let result = + activities::execute_http::execute(ctx, route.pool.clone(), config_json) + .await; + route.close().await; + result + } + }, + ) + .register( + activities::execute_multipart::NAME, + move |ctx: ActivityContext, config_json: String| { + let pool = multipart_pool.clone(); + async move { + let route = pool.route(ctx.instance_id()).await?; + let result = activities::execute_multipart::execute( + ctx, + route.pool.clone(), + config_json, + ) + .await; + route.close().await; + result } }, ) - .register(activities::update_instance_status::NAME, move |ctx: ActivityContext, input_json: String| { - let pool = status_pool.clone(); - async move { activities::update_instance_status::execute(ctx, pool, input_json).await } - }) - .register(activities::update_node_status::NAME, move |ctx: ActivityContext, input_json: String| { - let pool = node_status_pool.clone(); - async move { activities::update_node_status::execute(ctx, pool, input_json).await } - }) - .register(activities::execute_http::NAME, move |ctx: ActivityContext, config_json: String| { - let pool = http_pool.clone(); - async move { activities::execute_http::execute(ctx, pool, config_json).await } - }) - .register(activities::execute_multipart::NAME, move |ctx: ActivityContext, config_json: String| { - let pool = multipart_pool.clone(); - async move { activities::execute_multipart::execute(ctx, pool, config_json).await } - }) .build() } diff --git a/src/types.rs b/src/types.rs index c0237d10..faeda6a1 100644 --- a/src/types.rs +++ b/src/types.rs @@ -11,10 +11,12 @@ use percent_encoding::{utf8_percent_encode, NON_ALPHANUMERIC}; use serde::{Deserialize, Serialize}; use std::ffi::{CStr, CString}; use std::str::FromStr; -use std::sync::{Arc, OnceLock}; +use std::sync::Arc; use std::time::Duration; use uuid::Uuid; +pub(crate) use crate::client::backend_control_state; + pub(crate) const WORKER_MANAGEMENT_APPLICATION_NAME: &str = "pg_durable:worker:management"; pub(crate) const WORKER_POLL_APPLICATION_NAME: &str = "pg_durable:worker:poll"; pub(crate) const WORKER_DUROXIDE_APPLICATION_NAME: &str = "pg_durable:worker:duroxide"; @@ -274,9 +276,14 @@ pub async fn connect_as_user( pub(crate) async fn connect_as_user_for_new_transaction( user: &str, + origin_database: &str, ) -> Result { - connect_as_user_with_application_name(user, None, BACKEND_NEW_TRANSACTION_APPLICATION_NAME) - .await + connect_as_user_with_application_name( + user, + Some(origin_database), + BACKEND_NEW_TRANSACTION_APPLICATION_NAME, + ) + .await } async fn connect_as_user_with_application_name( @@ -369,13 +376,166 @@ fn resolve_duroxide_schema_spi() -> String { } } -/// Resolve the duroxide provider schema for the current backend session, -/// caching it for the session lifetime. The value cannot change without an -/// extension upgrade, which requires a reconnect to observe reliably, so a -/// per-session cache is safe. +fn backend_schema_name(schema: &str) -> Result<&'static str, String> { + match schema { + "_duroxide" => Ok("_duroxide"), + LEGACY_DUROXIDE_SCHEMA => Ok(LEGACY_DUROXIDE_SCHEMA), + _ => Err(format!("Unsupported duroxide provider schema: {schema}")), + } +} + +#[cfg(test)] +mod backend_schema_tests { + use super::{backend_control_connection_options, backend_schema_name}; + + #[test] + fn control_connection_bounds_server_queries_at_startup() { + let options = backend_control_connection_options( + "postgres://worker@localhost/control?options=-c%20statement_timeout%3D0", + ) + .unwrap(); + assert!(options + .get_options() + .unwrap() + .ends_with("-c statement_timeout=1500ms -c lock_timeout=1500ms")); + assert_eq!(options.get_username(), "worker"); + assert_eq!(options.get_database(), Some("control")); + } + + #[test] + fn schema_resolution_does_not_pin_a_previous_installation() { + assert_eq!(backend_schema_name("duroxide"), Ok("duroxide")); + assert_eq!(backend_schema_name("_duroxide"), Ok("_duroxide")); + assert_eq!(backend_schema_name("duroxide"), Ok("duroxide")); + } + + #[test] + fn rejects_unrecognized_schema_identifiers() { + for schema in ["", "df", "duroxide; SELECT 1", "\"_duroxide\""] { + assert!(backend_schema_name(schema).is_err()); + } + } +} + +/// Resolve without a lifetime cache: drop/recreate may change the provider schema. +/// Satellites resolve against the control database using the worker credential. pub fn backend_duroxide_schema() -> &'static str { - static SCHEMA: OnceLock = OnceLock::new(); - SCHEMA.get_or_init(resolve_duroxide_schema_spi) + try_backend_duroxide_schema().unwrap_or_else(|error| pgrx::error!("{error}")) +} + +pub(crate) fn try_backend_duroxide_schema() -> Result<&'static str, String> { + let current_database = Spi::get_one::("SELECT pg_catalog.current_database()::text") + .map_err(|error| format!("Failed to resolve caller database: {error}"))? + .ok_or("Failed to resolve caller database")?; + if current_database == get_database() { + backend_schema_name(&resolve_duroxide_schema_spi()) + } else { + backend_control_state(&postgres_connection_string()).map(|state| state.schema) + } +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct BackendControlState { + pub extension_oid: i64, + pub schema: &'static str, + pub ready: bool, +} + +pub(crate) fn backend_local_control_state() -> Result, String> { + let database = Spi::get_one::("SELECT pg_catalog.current_database()::text") + .map_err(|error| error.to_string())? + .ok_or("Caller database is unavailable")?; + if database != get_database() { + return Ok(None); + } + let extension_oid = Spi::get_one::( + "SELECT oid::bigint FROM pg_catalog.pg_extension WHERE extname = 'pg_durable'", + ) + .map_err(|error| error.to_string())? + .ok_or("pg_durable control installation unavailable")?; + let schema = backend_schema_name(&resolve_duroxide_schema_spi())?; + let table_exists = Spi::get_one::(&format!( + "SELECT EXISTS(SELECT 1 FROM pg_catalog.pg_tables + WHERE schemaname = '{schema}' AND tablename = '_worker_ready')" + )) + .map_err(|error| error.to_string())? + .unwrap_or(false); + let ready = table_exists + && Spi::get_one::(&format!( + "SELECT EXISTS(SELECT 1 FROM {schema}._worker_ready WHERE schema_version >= {})", + crate::WORKER_SCHEMA_VERSION + )) + .map_err(|error| error.to_string())? + .unwrap_or(false); + Ok(Some(BackendControlState { + extension_oid, + schema, + ready, + })) +} + +pub(crate) fn backend_control_connection_options( + database_url: &str, +) -> Result { + sqlx::postgres::PgConnectOptions::from_str(database_url) + .map(|options| { + options + .application_name(BACKEND_MONITORING_APPLICATION_NAME) + .options([("statement_timeout", "1500ms"), ("lock_timeout", "1500ms")]) + }) + .map_err(|error| format!("Invalid control database connection options: {error}")) +} + +pub(crate) async fn read_backend_control_state( + connection: &mut sqlx::PgConnection, +) -> Result { + let unavailable = |error| format!("pg_durable control installation unavailable: {error}"); + let (extension_oid, helper_exists, legacy_ready_exists, current_ready_exists) = + sqlx::query_as::<_, (i64, bool, bool, bool)>( + "SELECT e.oid::bigint, \ + EXISTS(SELECT 1 FROM pg_catalog.pg_proc p \ + JOIN pg_catalog.pg_namespace n ON n.oid = p.pronamespace \ + WHERE n.nspname = 'df' AND p.proname = 'duroxide_schema' AND p.pronargs = 0), \ + EXISTS(SELECT 1 FROM pg_catalog.pg_tables \ + WHERE schemaname = 'duroxide' AND tablename = '_worker_ready'), \ + EXISTS(SELECT 1 FROM pg_catalog.pg_tables \ + WHERE schemaname = '_duroxide' AND tablename = '_worker_ready') \ + FROM pg_catalog.pg_extension e WHERE e.extname = 'pg_durable'", + ) + .fetch_optional(&mut *connection) + .await + .map_err(unavailable)? + .ok_or_else(|| "pg_durable control installation unavailable".to_string())?; + let schema = if helper_exists { + let schema: String = sqlx::query_scalar("SELECT df.duroxide_schema()") + .fetch_one(&mut *connection) + .await + .map_err(unavailable)?; + backend_schema_name(&schema)? + } else { + LEGACY_DUROXIDE_SCHEMA + }; + let table_exists = if schema == LEGACY_DUROXIDE_SCHEMA { + legacy_ready_exists + } else { + current_ready_exists + }; + let ready = if table_exists { + sqlx::query_scalar(&format!( + "SELECT EXISTS(SELECT 1 FROM \"{schema}\"._worker_ready WHERE schema_version >= $1)" + )) + .bind(crate::WORKER_SCHEMA_VERSION) + .fetch_one(&mut *connection) + .await + .map_err(unavailable)? + } else { + false + }; + Ok(BackendControlState { + extension_oid, + schema, + ready, + }) } /// Resolve the duroxide provider schema name from the background worker using an diff --git a/src/worker.rs b/src/worker.rs index 1e5dd55c..6a827bcf 100644 --- a/src/worker.rs +++ b/src/worker.rs @@ -8,14 +8,18 @@ use pgrx::bgworkers::*; use pgrx::prelude::*; +use sqlx::Connection; +use std::collections::{BTreeMap, HashSet}; +use std::str::FromStr; use std::sync::Arc; use std::time::Duration; use duroxide::runtime; -use duroxide::{Client, InstanceFilter}; +use duroxide::{Client, ClientError, InstanceFilter}; use duroxide_pg::PostgresProvider; use tracing_subscriber::EnvFilter; +use crate::origin::{Origin, Router}; use crate::registry::{create_activity_registry, create_orchestration_registry}; use crate::types::{ get_max_duroxide_connections, get_max_management_connections, get_max_user_connections, @@ -319,8 +323,13 @@ async fn run_duroxide_runtime() { // Write the worker readiness record so backend sessions know the // duroxide schema is fully initialized for this schema version. // Skipped if the row already has the current WORKER_SCHEMA_VERSION. - if let Err(e) = write_worker_ready(&mgmt_pool, &duroxide_schema).await { + if let Err(e) = write_worker_ready(&mgmt_pool, &duroxide_schema, epoch_oid).await { log!("pg_durable: failed to write worker readiness record: {}", e); + teardown_runtime(duroxide_runtime, duroxide_store).await; + if !sleep_or_shutdown(STALE_RUNTIME_RETRY_INTERVAL).await { + break; + } + continue; } // Write a sentinel so we can detect drop+recreate even if the @@ -795,7 +804,42 @@ async fn write_epoch_sentinel(pool: &sqlx::PgPool) -> Result Result<(), sqlx::Error> { +async fn write_worker_ready( + pool: &sqlx::PgPool, + schema_name: &str, + epoch_oid: i64, +) -> Result<(), sqlx::Error> { + let mut transaction = pool.begin().await?; + sqlx::query("SET LOCAL lock_timeout = '1500ms'") + .execute(&mut *transaction) + .await?; + sqlx::query("LOCK TABLE df.instances, df.nodes IN ACCESS SHARE MODE") + .execute(&mut *transaction) + .await?; + let current_epoch: Option = sqlx::query_scalar( + "SELECT oid::bigint FROM pg_catalog.pg_extension WHERE extname = 'pg_durable'", + ) + .fetch_optional(&mut *transaction) + .await?; + if current_epoch != Some(epoch_oid) { + return Err(sqlx::Error::Protocol( + "Control installation changed before readiness publication".to_string(), + )); + } + let schema_name = format!("\"{}\"", schema_name.replace('"', "\"\"")); + sqlx::query(&format!( + "CREATE TABLE IF NOT EXISTS {schema_name}._origins ( + database_oid BIGINT NOT NULL, + installation_id UUID NOT NULL, + PRIMARY KEY (database_oid, installation_id) + )" + )) + .execute(&mut *transaction) + .await?; + sqlx::query(&format!("REVOKE ALL ON {schema_name}._origins FROM PUBLIC")) + .execute(&mut *transaction) + .await?; + sqlx::query(&format!( "CREATE TABLE IF NOT EXISTS {schema}._worker_ready ( sentinel BOOLEAN PRIMARY KEY DEFAULT TRUE, @@ -805,7 +849,7 @@ async fn write_worker_ready(pool: &sqlx::PgPool, schema_name: &str) -> Result<() )", schema = schema_name )) - .execute(pool) + .execute(&mut *transaction) .await?; // Allow non-superuser sessions to read the readiness record via @@ -814,13 +858,13 @@ async fn write_worker_ready(pool: &sqlx::PgPool, schema_name: &str) -> Result<() "GRANT USAGE ON SCHEMA {schema} TO PUBLIC", schema = schema_name )) - .execute(pool) + .execute(&mut *transaction) .await?; sqlx::query(&format!( "GRANT SELECT ON {schema}._worker_ready TO PUBLIC", schema = schema_name )) - .execute(pool) + .execute(&mut *transaction) .await?; sqlx::query(&format!( @@ -833,9 +877,25 @@ async fn write_worker_ready(pool: &sqlx::PgPool, schema_name: &str) -> Result<() schema = schema_name )) .bind(crate::WORKER_SCHEMA_VERSION) - .execute(pool) + .execute(&mut *transaction) .await?; + transaction.commit().await?; + Ok(()) +} + +pub(crate) async fn register_origin(pool: &sqlx::PgPool, origin: &Origin) -> Result<(), String> { + let schema = resolve_duroxide_schema_pool(pool).await; + let schema = format!("\"{}\"", schema.replace('"', "\"\"")); + sqlx::query(&format!( + "INSERT INTO {schema}._origins (database_oid, installation_id) + VALUES ($1, $2) ON CONFLICT (database_oid, installation_id) DO NOTHING" + )) + .bind(i64::from(origin.database_oid)) + .bind(origin.installation_id) + .execute(pool) + .await + .map_err(|error| format!("register origin: {error}"))?; Ok(()) } @@ -864,6 +924,7 @@ async fn select_expired_instance_ids_tx( tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, retention_days: i32, max_keep: i64, + limit: Option, ) -> Result, sqlx::Error> { let ids: Option> = sqlx::query_scalar( r#" @@ -894,14 +955,19 @@ async fn select_expired_instance_ids_tx( -- days). Retained rows are thus always within the newest $1 AND younger than -- the retention window, so the retained terminal count never exceeds $1. SELECT pg_catalog.array_agg(id) - FROM terminal_instances - WHERE terminal_rank OPERATOR(pg_catalog.>) $1 - OR terminal_at OPERATOR(pg_catalog.<) - (pg_catalog.now() OPERATOR(pg_catalog.-) pg_catalog.make_interval(days => $2::int)) + FROM ( + SELECT id FROM terminal_instances + WHERE terminal_rank OPERATOR(pg_catalog.>) $1 + OR terminal_at OPERATOR(pg_catalog.<) + (pg_catalog.now() OPERATOR(pg_catalog.-) pg_catalog.make_interval(days => $2::int)) + ORDER BY terminal_rank DESC + LIMIT $3 + ) expired "#, ) .bind(max_keep) .bind(retention_days) + .bind(limit) .fetch_one(&mut **tx) .await?; Ok(ids.unwrap_or_default()) @@ -962,7 +1028,7 @@ async fn select_expired_instance_ids( max_keep: i64, ) -> Result, sqlx::Error> { let mut tx = pool.begin().await?; - let ids = select_expired_instance_ids_tx(&mut tx, retention_days, max_keep).await?; + let ids = select_expired_instance_ids_tx(&mut tx, retention_days, max_keep, None).await?; tx.commit().await?; Ok(ids) } @@ -988,7 +1054,7 @@ pub(crate) async fn delete_expired_instances_transaction( retention_days: i32, max_keep: i64, ) -> Result { - let ids = select_expired_instance_ids_tx(tx, retention_days, max_keep).await?; + let ids = select_expired_instance_ids_tx(tx, retention_days, max_keep, None).await?; delete_expired_instances_tx(tx, &ids).await } @@ -1004,6 +1070,9 @@ async fn run_until_extension_dropped_or_shutdown( log!("pg_durable: processing durable functions..."); let client = Client::new(duroxide_store.clone()); + let router = Router::new(Arc::new(maintenance_pool.clone())); + let mut origin_cursor = OriginRetentionCursor::default(); + let mut engine_cursor = String::new(); let mut drop_check = tokio::time::interval(drop_poll_interval); drop_check.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); @@ -1023,7 +1092,7 @@ async fn run_until_extension_dropped_or_shutdown( ); reconcile_check.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); - loop { + 'processing: loop { tokio::select! { _ = tokio::time::sleep(shutdown_check_interval) => { // is_shutdown_requested reads a volatile atomic; no spawn_blocking needed. @@ -1043,6 +1112,7 @@ async fn run_until_extension_dropped_or_shutdown( } } _ = reconcile_check.tick(), if reconcile_enabled => { + let maintenance = async { let retention_days = get_retention_days(); // Engine-first: retire the engine record before the df row, so a @@ -1088,6 +1158,37 @@ async fn run_until_extension_dropped_or_shutdown( Ok(_) => {} Err(e) => log!("pg_durable: reclaiming orphaned engine records failed: {e}"), } + + let schema = resolve_duroxide_schema_pool(maintenance_pool).await; + let schema = format!("\"{}\"", schema.replace('"', "\"\"")); + if let Err(error) = sweep_registered_origins( + maintenance_pool, &client, &router, &schema, retention_days, &mut origin_cursor, + ).await { + log!("pg_durable: origin retention failed: {error}"); + } + if let Err(error) = reclaim_origin_instances( + maintenance_pool, &client, &router, &schema, retention, &mut engine_cursor, + ).await { + log!("pg_durable: origin reconciliation failed: {error}"); + } + }; + tokio::pin!(maintenance); + loop { + tokio::select! { + _ = &mut maintenance => break, + _ = wait_for_shutdown() => break 'processing, + _ = drop_check.tick() => { + let still_valid = match epoch_id { + Some(eid) => check_epoch_sentinel(poll_pool, eid).await, + None => check_extension_exists(poll_pool).await, + }; + if !still_valid { + log!("pg_durable: control extension removed during maintenance"); + break 'processing; + } + } + } + } } } } @@ -1095,6 +1196,451 @@ async fn run_until_extension_dropped_or_shutdown( teardown_runtime(duroxide_runtime, duroxide_store).await; } +const ORIGIN_BATCH: i64 = 8; +const ORIGIN_ENGINE_BATCH: i64 = 100; +const ORIGIN_OPERATION_TIMEOUT: Duration = Duration::from_secs(15); + +#[derive(Default)] +struct OriginRetentionCursor { + origin: (i64, uuid::Uuid), + after_id: Option, +} + +impl OriginRetentionCursor { + fn enter(&mut self, origin: (i64, uuid::Uuid)) { + if self.origin != origin { + self.after_id = None; + } + self.origin = origin; + } + + fn resume_after_pass(&mut self, previous_id: Option) -> bool { + if self.after_id.is_some() && self.after_id != previous_id { + return true; + } + self.after_id = None; + false + } +} + +#[derive(sqlx::FromRow)] +struct OriginRetentionCandidate { + id: String, + terminal_rank: i64, + expired_by_age: bool, +} + +impl OriginRetentionCandidate { + fn is_expired(&self, max_keep: i64) -> bool { + self.terminal_rank > max_keep || self.expired_by_age + } +} + +async fn retire_origin_candidates( + candidates: Vec, + after_id: &mut Option, + max_keep: i64, + mut retire: Retire, +) -> Result<(), String> +where + Retire: FnMut(String) -> Retired, + Retired: std::future::Future>, +{ + if candidates.is_empty() { + *after_id = None; + } + for candidate in candidates { + *after_id = Some(candidate.id.clone()); + if candidate.is_expired(max_keep) { + retire(candidate.id).await?; + } + } + Ok(()) +} + +async fn sweep_registered_origins( + pool: &sqlx::PgPool, + client: &Client, + router: &Router, + schema: &str, + retention_days: i32, + cursor: &mut OriginRetentionCursor, +) -> Result<(), String> { + let origins: Vec<(i64, uuid::Uuid)> = sqlx::query_as(&format!( + "SELECT database_oid, installation_id FROM {schema}._origins + WHERE (database_oid, installation_id) > ($1, $2) + OR ((database_oid, installation_id) = ($1, $2) AND $4) + ORDER BY database_oid, installation_id LIMIT $3" + )) + .bind(cursor.origin.0) + .bind(cursor.origin.1) + .bind(ORIGIN_BATCH) + .bind(cursor.after_id.is_some()) + .fetch_all(pool) + .await + .map_err(|error| format!("list registered origins: {error}"))?; + if origins.is_empty() { + *cursor = OriginRetentionCursor::default(); + } + for (database_oid, installation_id) in origins { + cursor.enter((database_oid, installation_id)); + let origin = Origin { + database_oid: u32::try_from(database_oid) + .map_err(|_| "Invalid registered database OID")?, + installation_id, + }; + let previous_id = cursor.after_id.clone(); + let result = tokio::time::timeout(ORIGIN_OPERATION_TIMEOUT, async { + let route = router.connect(&origin).await?; + let result = retire_origin_instances( + &route.pool, + client, + &origin, + retention_days, + &mut cursor.after_id, + ) + .await; + route.close().await; + result + }) + .await; + match result { + Ok(Ok(())) => {} + Ok(Err(error)) => log!("pg_durable: retention for origin {origin:?} deferred: {error}"), + Err(_) => log!("pg_durable: retention for origin {origin:?} timed out"), + } + if cursor.resume_after_pass(previous_id) { + break; + } + } + Ok(()) +} + +async fn retire_origin_instances( + pool: &sqlx::PgPool, + client: &Client, + origin: &Origin, + retention_days: i32, + after_id: &mut Option, +) -> Result<(), String> { + let mut tx = pool.begin().await.map_err(|error| error.to_string())?; + let candidates: Vec = sqlx::query_as( + r#" + WITH terminal_instances AS ( + SELECT id, COALESCE(completed_at, created_at) AS terminal_at, + pg_catalog.row_number() OVER ( + ORDER BY COALESCE(completed_at, created_at) DESC NULLS LAST, id DESC + ) AS terminal_rank + FROM df.instances + WHERE status OPERATOR(pg_catalog.=) ANY (ARRAY['completed', 'failed', 'cancelled']) + ) + SELECT id, terminal_rank, + terminal_at OPERATOR(pg_catalog.<) + (pg_catalog.now() OPERATOR(pg_catalog.-) pg_catalog.make_interval(days => $1::int)) + AS expired_by_age + FROM terminal_instances + WHERE $2::text IS NULL OR id OPERATOR(pg_catalog.>) $2 + ORDER BY id LIMIT $3 + "#, + ) + .bind(retention_days) + .bind(after_id.as_deref()) + .bind(i64::from(RECLAIM_BATCH)) + .fetch_all(&mut *tx) + .await + .map_err(|error| error.to_string())?; + tx.commit().await.map_err(|error| error.to_string())?; + retire_origin_candidates( + candidates, + after_id, + TERMINAL_INSTANCE_MAX_KEEP, + |local_id| async move { + let engine_id = origin.engine_id(&local_id); + if origin_local_id(origin, &engine_id).is_none() { + return Ok(()); + } + match client.delete_instance(&engine_id, false).await { + Ok(_) | Err(ClientError::InstanceNotFound { .. }) => { + delete_expired_instances(pool, &[local_id]) + .await + .map_err(|error| error.to_string())?; + } + Err(ClientError::InstanceStillRunning { .. }) => {} + Err(error) => return Err(format!("retire origin engine record: {error}")), + } + Ok(()) + }, + ) + .await +} + +fn origin_local_id<'a>(origin: &Origin, engine_id: &'a str) -> Option<&'a str> { + if engine_id.contains("::") + || Origin::from_engine_id(engine_id).ok().flatten().as_ref() != Some(origin) + { + return None; + } + engine_id.strip_prefix(&origin.engine_id("")) +} + +fn select_origin_orphans( + origin: &Origin, + failed_ids: Vec, + present_local_ids: &HashSet, +) -> Vec { + failed_ids + .into_iter() + .filter(|engine_id| { + origin_local_id(origin, engine_id) + .is_some_and(|local_id| !present_local_ids.contains(local_id)) + }) + .collect() +} + +async fn reclaim_origin_instances( + pool: &sqlx::PgPool, + client: &Client, + router: &Router, + schema: &str, + retention: Duration, + cursor: &mut String, +) -> Result<(), String> { + let candidates: Vec<(String, String)> = sqlx::query_as(&format!( + "SELECT i.instance_id, e.status FROM {schema}.instances i + JOIN {schema}.executions e ON e.instance_id = i.instance_id + AND e.execution_id = i.current_execution_id + WHERE i.instance_id > $1 AND i.parent_instance_id IS NULL + AND i.instance_id NOT LIKE '%::%' + AND EXISTS (SELECT 1 FROM {schema}._origins o + WHERE i.instance_id LIKE 'pgdf-' || o.database_oid::text || '-' || + pg_catalog.replace(o.installation_id::text, '-', '') || '-%') + ORDER BY i.instance_id LIMIT $2" + )) + .bind(cursor.as_str()) + .bind(ORIGIN_ENGINE_BATCH) + .fetch_all(pool) + .await + .map_err(|error| format!("list registered origin engine roots: {error}"))?; + if let Some((last_id, _)) = candidates.last() { + *cursor = last_id.clone(); + } else { + cursor.clear(); + } + let mut by_origin = BTreeMap::<(u32, uuid::Uuid), Vec<(String, String)>>::new(); + for (engine_id, status) in candidates { + if let Ok(Some(origin)) = Origin::from_engine_id(&engine_id) { + by_origin + .entry((origin.database_oid, origin.installation_id)) + .or_default() + .push((engine_id, status)); + } + } + for ((database_oid, installation_id), records) in by_origin { + let origin = Origin { + database_oid, + installation_id, + }; + let result = tokio::time::timeout(ORIGIN_OPERATION_TIMEOUT, async { + match router.connect(&origin).await { + Ok(route) => { + let result = + reclaim_existing_origin(&route.pool, client, &origin, records, retention) + .await; + route.close().await; + result + } + Err(route_error) => { + if origin_is_removed(pool, &origin).await? { + reclaim_removed_origin(client, &origin, records, retention).await + } else { + Err(route_error) + } + } + } + }) + .await; + match result { + Ok(Ok(reclaimed)) if reclaimed > 0 => { + log!("pg_durable: reclaimed {reclaimed} engine record(s) for origin {origin:?}"); + } + Ok(Ok(_)) => {} + Ok(Err(error)) => { + log!("pg_durable: reconciliation for origin {origin:?} deferred: {error}") + } + Err(_) => log!("pg_durable: reconciliation for origin {origin:?} timed out"), + } + } + Ok(()) +} + +async fn reclaim_existing_origin( + pool: &sqlx::PgPool, + client: &Client, + origin: &Origin, + records: Vec<(String, String)>, + retention: Duration, +) -> Result { + let failed: Vec = records + .into_iter() + .filter(|(_, status)| status == "Failed") + .map(|(engine_id, _)| engine_id) + .collect(); + let local_ids: Vec<&str> = failed + .iter() + .filter_map(|engine_id| origin_local_id(origin, engine_id)) + .collect(); + if local_ids.is_empty() { + return Ok(0); + } + let present: HashSet = + sqlx::query_scalar("SELECT id FROM df.instances WHERE id = ANY($1)") + .bind(&local_ids) + .fetch_all(pool) + .await + .map_err(|error| format!("cross-check origin df.instances: {error}"))? + .into_iter() + .collect(); + let orphans = select_origin_orphans(origin, failed, &present); + if orphans.is_empty() { + return Ok(0); + } + client + .delete_instance_bulk(InstanceFilter { + instance_ids: Some(orphans), + completed_before: Some(retention_cutoff_ms(retention)), + limit: Some(RECLAIM_BATCH), + }) + .await + .map(|result| result.instances_deleted) + .map_err(|error| format!("delete origin orphans: {error}")) +} + +async fn reclaim_removed_origin( + client: &Client, + origin: &Origin, + records: Vec<(String, String)>, + retention: Duration, +) -> Result { + let mut roots = Vec::new(); + for (engine_id, status) in records { + if origin_local_id(origin, &engine_id).is_none() { + continue; + } + if status == "Running" { + client + .cancel_instance(&engine_id, "pg_durable origin installation removed") + .await + .map_err(|error| format!("cancel removed origin root: {error}"))?; + } + roots.push(engine_id); + } + if roots.is_empty() { + return Ok(0); + } + client + .delete_instance_bulk(InstanceFilter { + instance_ids: Some(roots), + completed_before: Some(retention_cutoff_ms(retention)), + limit: Some(RECLAIM_BATCH), + }) + .await + .map(|result| result.instances_deleted) + .map_err(|error| format!("delete removed origin roots: {error}")) +} + +async fn origin_is_removed(pool: &sqlx::PgPool, origin: &Origin) -> Result { + let database: Option = sqlx::query_scalar( + "SELECT datname FROM pg_catalog.pg_database WHERE oid = $1::bigint::oid", + ) + .bind(i64::from(origin.database_oid)) + .fetch_optional(pool) + .await + .map_err(|error| format!("probe origin database: {error}"))?; + let Some(database) = database else { + return Ok(true); + }; + let _permit = crate::origin::acquire_connections(1).await?; + let options = sqlx::postgres::PgConnectOptions::from_str( + &postgres_connection_string_with_application_name(WORKER_MANAGEMENT_APPLICATION_NAME), + ) + .map_err(|error| error.to_string())? + .database(&database); + let mut probe = tokio::time::timeout( + Duration::from_secs(5), + sqlx::PgConnection::connect_with(&options), + ) + .await + .map_err(|_| "Origin absence connection timed out".to_string())? + .map_err(|error| format!("connect origin absence probe: {error}"))?; + let result = probe_origin_installation(&mut probe, origin).await; + probe + .close() + .await + .map_err(|error| format!("close origin absence probe: {error}"))?; + result +} + +async fn probe_origin_installation( + connection: &mut sqlx::PgConnection, + origin: &Origin, +) -> Result { + let mut tx = connection + .begin() + .await + .map_err(|error| error.to_string())?; + sqlx::query("SET LOCAL lock_timeout = '1500ms'") + .execute(&mut *tx) + .await + .map_err(|error| error.to_string())?; + sqlx::query("SET LOCAL statement_timeout = '5s'") + .execute(&mut *tx) + .await + .map_err(|error| error.to_string())?; + let database_oid: i64 = sqlx::query_scalar( + "SELECT oid::bigint FROM pg_catalog.pg_database + WHERE datname = pg_catalog.current_database()", + ) + .fetch_one(&mut *tx) + .await + .map_err(|error| error.to_string())?; + if database_oid != i64::from(origin.database_oid) { + return Err("Origin database changed during absence probe".to_string()); + } + let owned: bool = sqlx::query_scalar( + "SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_extension e + JOIN pg_catalog.pg_depend d ON d.refobjid = e.oid + AND d.refclassid = 'pg_catalog.pg_extension'::regclass AND d.deptype = 'e' + JOIN pg_catalog.pg_class c ON c.oid = d.objid + AND d.classid = 'pg_catalog.pg_class'::regclass + JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace + WHERE e.extname = 'pg_durable' AND n.nspname = 'df' AND c.relname = '_installation')", + ) + .fetch_one(&mut *tx) + .await + .map_err(|error| error.to_string())?; + if !owned { + return Ok(true); + } + sqlx::query("LOCK TABLE df._installation IN ACCESS SHARE MODE") + .execute(&mut *tx) + .await + .map_err(|error| error.to_string())?; + let present: bool = sqlx::query_scalar( + "SELECT EXISTS (SELECT 1 FROM df._installation i + JOIN pg_catalog.pg_depend d ON d.objid = 'df._installation'::regclass + AND d.classid = 'pg_catalog.pg_class'::regclass AND d.deptype = 'e' + AND d.refclassid = 'pg_catalog.pg_extension'::regclass + JOIN pg_catalog.pg_extension e ON e.oid = d.refobjid AND e.extname = 'pg_durable' + WHERE i.id = $1)", + ) + .bind(origin.installation_id) + .fetch_one(&mut *tx) + .await + .map_err(|error| error.to_string())?; + tx.commit().await.map_err(|error| error.to_string())?; + Ok(!present) +} + /// Shut down a duroxide runtime and close its store pool. /// /// Two callers, and the branch below distinguishes them by cause rather than by @@ -1174,12 +1720,14 @@ fn retention_cutoff_ms(retention: Duration) -> u64 { /// df.instances row, select the orphans to reclaim: those with no df row and that /// are not sub-orchestrations. Both legacy engine-named children and current /// explicitly named composed children legitimately have no df row and must be kept. +/// Namespaced IDs require a separate cross-check in their registered origin. pub(crate) fn select_orphans( failed_ids: Vec, present: &std::collections::HashSet, ) -> Vec { failed_ids .into_iter() + .filter(|id| matches!(crate::origin::Origin::from_engine_id(id), Ok(None))) .filter(|id| !is_sub_orchestration(id) && !present.contains(id)) .collect() } @@ -1274,3 +1822,267 @@ async fn retire_engine_records(client: &Client, ids: &[String]) -> bool { } } } + +#[cfg(test)] +mod tests { + use super::*; + use crate::origin::Origin; + use std::collections::HashSet; + + #[test] + fn worker_origin_retention_advances_past_running_and_undecidable_prefix() { + tokio::runtime::Builder::new_current_thread() + .enable_time() + .build() + .unwrap() + .block_on(async { + for undecidable_prefix in [false, true] { + let mut after_id = None; + let mut deleted = Vec::new(); + let mut attempts = Vec::new(); + let target = format!("{:08x}", RECLAIM_BATCH + 1); + let mut passes = 0; + while after_id.as_deref() != Some(target.as_str()) { + let candidates: Vec<_> = (u32::from(!undecidable_prefix) + ..=RECLAIM_BATCH + 1) + .map(|index| OriginRetentionCandidate { + id: format!("{index:08x}"), + terminal_rank: i64::from(RECLAIM_BATCH + 2 - index), + expired_by_age: true, + }) + .filter(|candidate| { + after_id.as_ref().is_none_or(|last| candidate.id > *last) + }) + .take(RECLAIM_BATCH as usize) + .collect(); + assert!(candidates.len() <= RECLAIM_BATCH as usize); + let _ = retire_origin_candidates(candidates, &mut after_id, 10, |id| { + attempts.push(id.clone()); + let result = if id == "00000000" { + Err("engine state undecidable".to_string()) + } else { + if id == target { + deleted.push(id); + } + Ok(()) + }; + std::future::ready(result) + }) + .await; + passes += 1; + assert!( + passes <= 3, + "retention must not restart at the skipped prefix" + ); + } + assert_eq!(passes, if undecidable_prefix { 3 } else { 2 }); + assert_eq!( + attempts.len(), + RECLAIM_BATCH as usize + 1 + usize::from(undecidable_prefix) + ); + assert_eq!(deleted, vec![target]); + } + }); + } + + #[test] + fn worker_origin_retention_timeout_keeps_candidate_progress() { + tokio::runtime::Builder::new_current_thread() + .enable_time() + .build() + .unwrap() + .block_on(async { + let mut after_id = None; + let candidates = vec![OriginRetentionCandidate { + id: "00000001".to_string(), + terminal_rank: 1, + expired_by_age: true, + }]; + assert!(tokio::time::timeout( + Duration::ZERO, + retire_origin_candidates(candidates, &mut after_id, 10, |_| { + std::future::pending::>() + }), + ) + .await + .is_err()); + assert_eq!(after_id.as_deref(), Some("00000001")); + + let mut cursor = OriginRetentionCursor { + origin: (42, uuid::Uuid::from_u128(7)), + after_id, + }; + assert!(cursor.resume_after_pass(None)); + let mut retired = Vec::new(); + let candidates = ["00000001", "00000002"] + .into_iter() + .filter(|id| Some(*id) > cursor.after_id.as_deref()) + .map(|id| OriginRetentionCandidate { + id: id.to_string(), + terminal_rank: 1, + expired_by_age: true, + }) + .collect(); + retire_origin_candidates(candidates, &mut cursor.after_id, 10, |id| { + retired.push(id); + std::future::ready(Ok(())) + }) + .await + .unwrap(); + assert_eq!(retired, vec!["00000002"]); + + retire_origin_candidates( + Vec::new(), + &mut cursor.after_id, + 10, + |_| -> std::future::Ready> { + panic!("an exhausted page cannot retire an instance"); + }, + ) + .await + .unwrap(); + assert!(!cursor.resume_after_pass(Some("00000002".to_string()))); + assert_eq!(cursor.after_id, None); + let candidates = vec![OriginRetentionCandidate { + id: "00000001".to_string(), + terminal_rank: 1, + expired_by_age: true, + }]; + retire_origin_candidates(candidates, &mut cursor.after_id, 10, |id| { + retired.push(id); + std::future::ready(Ok(())) + }) + .await + .unwrap(); + assert_eq!(retired, vec!["00000002", "00000001"]); + }); + } + + #[test] + fn worker_origin_retention_preserves_max_keep_and_age() { + tokio::runtime::Builder::new_current_thread() + .build() + .unwrap() + .block_on(async { + let candidates = [(1, false), (10, false), (11, false), (2, true), (12, true)] + .into_iter() + .enumerate() + .map( + |(index, (terminal_rank, expired_by_age))| OriginRetentionCandidate { + id: format!("{index:08x}"), + terminal_rank, + expired_by_age, + }, + ) + .collect(); + let mut after_id = None; + let mut retired = Vec::new(); + retire_origin_candidates(candidates, &mut after_id, 10, |id| { + retired.push(id); + std::future::ready(Ok(())) + }) + .await + .unwrap(); + assert_eq!(retired, vec!["00000002", "00000003", "00000004"]); + assert_eq!(after_id.as_deref(), Some("00000004")); + }); + } + + #[test] + fn worker_origin_retention_cursor_is_scoped_to_installation() { + let first = (42, uuid::Uuid::from_u128(7)); + let replacement = (42, uuid::Uuid::from_u128(8)); + let mut cursor = OriginRetentionCursor::default(); + cursor.enter(first); + cursor.after_id = Some("deadbeef".to_string()); + cursor.enter(first); + assert_eq!(cursor.after_id.as_deref(), Some("deadbeef")); + cursor.enter(replacement); + assert_eq!(cursor.after_id, None); + cursor.after_id = Some("cafebabe".to_string()); + cursor.enter((43, uuid::Uuid::from_u128(8))); + assert_eq!(cursor.after_id, None); + cursor.after_id = Some("deadbeef".to_string()); + assert!(!cursor.resume_after_pass(Some("deadbeef".to_string()))); + assert_eq!(cursor.after_id, None); + } + + #[test] + fn worker_legacy_orphans_exclude_satellites_and_malformed_namespaces() { + let origin = Origin { + database_oid: 42, + installation_id: uuid::Uuid::from_u128(7), + }; + let failed = vec![ + "deadbeef".to_string(), + "cafebabe".to_string(), + origin.engine_id("deadbeef"), + format!("{}::2::cafebabe", origin.engine_id("deadbeef")), + "pgdf-invalid".to_string(), + "sub::child".to_string(), + "deadbeef::sub::child".to_string(), + "deadbeef::2::cafebabe".to_string(), + ]; + assert_eq!( + select_orphans(failed, &HashSet::from(["cafebabe".to_string()])), + vec!["deadbeef".to_string()] + ); + } + + #[test] + fn worker_origin_orphans_require_matching_database_and_installation() { + let origin = Origin { + database_oid: 42, + installation_id: uuid::Uuid::from_u128(7), + }; + let other_database = Origin { + database_oid: 43, + ..origin.clone() + }; + let replacement = Origin { + installation_id: uuid::Uuid::from_u128(8), + ..origin.clone() + }; + let candidates = vec![ + origin.engine_id("deadbeef"), + origin.engine_id("cafebabe"), + other_database.engine_id("deadbeef"), + replacement.engine_id("deadbeef"), + "deadbeef".to_string(), + "pgdf-invalid".to_string(), + format!("{}::2::12345678", origin.engine_id("deadbeef")), + format!("{}::2::12345678::3::abcdef12", origin.engine_id("deadbeef")), + ]; + assert_eq!( + select_origin_orphans( + &origin, + candidates, + &HashSet::from(["cafebabe".to_string()]) + ), + vec![origin.engine_id("deadbeef")] + ); + } + + #[test] + fn worker_origin_local_id_accepts_only_canonical_roots() { + let origin = Origin { + database_oid: 42, + installation_id: uuid::Uuid::from_u128(7), + }; + let root = origin.engine_id("deadbeef"); + assert_eq!(origin_local_id(&origin, &root), Some("deadbeef")); + assert_eq!( + origin_local_id(&origin, &format!("{root}::1::cafebabe")), + None + ); + assert_eq!( + origin_local_id(&origin, &origin.engine_id("invalid!")), + None + ); + assert_eq!(origin_local_id(&origin, "deadbeef"), None); + assert_eq!( + origin_local_id(&origin, &root.replacen("pgdf-42-", "pgdf-042-", 1)), + None + ); + } +} diff --git a/tests/e2e/sql/14_database.sql b/tests/e2e/sql/14_database.sql index 33e90379..c09cd7c6 100644 --- a/tests/e2e/sql/14_database.sql +++ b/tests/e2e/sql/14_database.sql @@ -2,10 +2,10 @@ -- Licensed under the PostgreSQL License. -- Merged from: 29_database_validation, 34_multi_database --- Tests: CREATE EXTENSION rejected in wrong database, workflows execute in correct database, +-- Tests: control and satellite installation, local APIs and transaction boundaries, -- df.start() with explicit database parameter, invalid database rejection, -- multi-node sequence in another database, dropped database failure handling --- Runs as postgres throughout (creates/drops databases) +-- Database administration runs as postgres; normal workflows use df_e2e_user. -- === Test: 29_database_validation === @@ -29,6 +29,35 @@ END $$; -- Test 1: CREATE EXTENSION should succeed in the correct database SELECT public._e2e_drop_extension_safe(); + +DROP DATABASE IF EXISTS _test_satellite_db WITH (FORCE); +CREATE DATABASE _test_satellite_db; + +DO $$ +DECLARE + connstr TEXT := format('host=localhost dbname=_test_satellite_db port=%s user=postgres', current_setting('port')); + installed BOOLEAN := false; + schema_count INT; +BEGIN + BEGIN + PERFORM dblink_exec(connstr, 'CREATE EXTENSION pg_durable'); + installed := true; + EXCEPTION WHEN OTHERS THEN + IF SQLERRM NOT ILIKE '%control%' THEN + RAISE EXCEPTION 'TEST FAILED [control absent]: unexpected install error: %', SQLERRM; + END IF; + END; + IF installed THEN + RAISE EXCEPTION 'TEST FAILED: satellite installed without a control installation'; + END IF; + SELECT total INTO schema_count FROM dblink(connstr, + 'SELECT count(*) FROM pg_namespace WHERE nspname IN (''df'', ''_duroxide'', ''duroxide'')' + ) AS remote(total INT); + IF schema_count IS DISTINCT FROM 0 THEN + RAISE EXCEPTION 'TEST FAILED: rejected installation left schemas behind'; + END IF; +END $$; + CREATE EXTENSION pg_durable; SELECT df.grant_usage('df_e2e_user'); @@ -44,6 +73,7 @@ BEGIN END $$; -- Test 2: Verify workflows can execute (BGW is connected to this database) +SET SESSION AUTHORIZATION df_e2e_user; CREATE TEMP TABLE _test_state (instance_id TEXT); INSERT INTO _test_state SELECT df.start('SELECT 42 as answer', 'test-correct-db'); @@ -65,40 +95,295 @@ BEGIN END $$; DROP TABLE _test_state; +RESET SESSION AUTHORIZATION; --- Test 3: CREATE EXTENSION must fail in a wrong database -DROP DATABASE IF EXISTS _test_wrong_db; -CREATE DATABASE _test_wrong_db; +-- Test 3: A satellite owns local metadata and uses the control runtime. +CREATE TEMP TABLE _control_epoch AS SELECT epoch_id FROM df._worker_epoch; +DROP TABLE IF EXISTS public.test_satellite_log; +CREATE TABLE public.test_satellite_log ( + marker TEXT PRIMARY KEY, + value INT DEFAULT 0, + db_name TEXT DEFAULT current_database(), + role_name TEXT DEFAULT current_user +); +GRANT SELECT, INSERT, UPDATE ON public.test_satellite_log TO df_e2e_user; +SELECT df.grant_usage('df_e2e_user', include_http => true); + +SELECT dblink_connect('satellite', format( + 'host=localhost dbname=_test_satellite_db port=%s user=postgres', current_setting('port') +)); +SELECT dblink_exec('satellite', 'CREATE EXTENSION pg_durable'); +SELECT dblink_exec('satellite', $remote$ + DO $check$ + BEGIN + IF EXISTS (SELECT 1 FROM pg_namespace WHERE nspname IN ('_duroxide', 'duroxide')) THEN + RAISE EXCEPTION 'TEST FAILED: satellite created a provider schema'; + END IF; + IF NOT EXISTS ( + SELECT 1 FROM pg_class AS relation + JOIN pg_extension AS extension ON extension.extname = 'pg_durable' + JOIN pg_depend AS dependency ON dependency.objid = relation.oid + AND dependency.classid = 'pg_class'::regclass + AND dependency.refclassid = 'pg_extension'::regclass + AND dependency.refobjid = extension.oid AND dependency.deptype = 'e' + WHERE relation.oid = 'df._installation'::regclass + AND relation.relowner = extension.extowner + ) THEN + RAISE EXCEPTION 'TEST FAILED: installation identity is not extension-owned'; + END IF; + PERFORM df.grant_usage('df_e2e_user'); + END $check$; + CREATE TABLE public.test_satellite_log ( + marker TEXT PRIMARY KEY, + value INT DEFAULT 0, + db_name TEXT DEFAULT current_database(), + role_name TEXT DEFAULT current_user + ); + GRANT SELECT, INSERT, UPDATE ON public.test_satellite_log TO df_e2e_user; + SET SESSION AUTHORIZATION df_e2e_user; + CREATE TEMP TABLE _satellite_state (name TEXT PRIMARY KEY, instance_id TEXT); +$remote$); DO $$ DECLARE - connstr TEXT; - err_msg TEXT; + satellite_id UUID; + control_id UUID; BEGIN - connstr := format( - 'host=localhost dbname=_test_wrong_db port=%s user=postgres', - current_setting('port') + SELECT id INTO STRICT control_id FROM df._installation WHERE singleton; + SELECT id INTO STRICT satellite_id FROM dblink('satellite', + 'SELECT id FROM df._installation WHERE singleton') AS remote(id UUID); + IF satellite_id IS NULL OR satellite_id = control_id THEN + RAISE EXCEPTION 'TEST FAILED: installations must have distinct non-null identities'; + END IF; +END $$; + +SELECT dblink_exec('satellite', $remote$ + DO $check$ BEGIN PERFORM df.setvar('satellite_value', '42'); END $check$; +$remote$); + +SELECT dblink_exec('satellite', $remote$ + DO $check$ + BEGIN + IF (SELECT count(*) FROM df._installation) <> 1 OR + has_table_privilege(current_user, 'df._installation', 'INSERT,UPDATE,DELETE,TRUNCATE,REFERENCES,TRIGGER') THEN + RAISE EXCEPTION 'TEST FAILED: installation identity must be a read-only singleton'; + END IF; + IF has_function_privilege(current_user, 'df.http(text,text,text,jsonb,integer)', 'EXECUTE') THEN + RAISE EXCEPTION 'TEST FAILED: control HTTP grant leaked into satellite'; + END IF; + END $check$; + INSERT INTO _satellite_state SELECT 'native', df.start( + 'INSERT INTO public.test_satellite_log (marker, value) VALUES (''native'', {satellite_value})' + ~> 'SELECT current_database()', 'test-satellite-native' ); +$remote$); +SELECT dblink_exec('satellite', $remote$ + DO $check$ + DECLARE + inst_id TEXT := (SELECT instance_id FROM _satellite_state WHERE name = 'native'); + status TEXT; BEGIN - PERFORM dblink_exec(connstr, 'CREATE EXTENSION pg_durable;'); - RAISE EXCEPTION 'TEST FAILED: CREATE EXTENSION should have been rejected in wrong database'; - EXCEPTION WHEN OTHERS THEN - err_msg := SQLERRM; - END; + status := df.await_instance(inst_id, 30); + IF status IS DISTINCT FROM 'completed' OR df.status(inst_id) IS DISTINCT FROM 'completed' THEN + RAISE EXCEPTION 'TEST FAILED [satellite native]: status = %', status; + END IF; + IF inst_id !~ '^[0-9a-f]{8}$' OR + COALESCE(position('_test_satellite_db' IN df.result(inst_id)), 0) = 0 OR + COALESCE(length(df.explain(inst_id)), 0) = 0 THEN + RAISE EXCEPTION 'TEST FAILED: satellite public ID/result/explain'; + END IF; + IF NOT EXISTS (SELECT 1 FROM df.instance_info(inst_id) AS info + WHERE info.instance_id = inst_id AND lower(info.status) = 'completed') OR + NOT EXISTS (SELECT 1 FROM df.list_instances() AS info + WHERE info.instance_id = inst_id AND lower(info.status) = 'completed') THEN + RAISE EXCEPTION 'TEST FAILED: satellite info/list did not resolve local instance'; + END IF; + IF NOT EXISTS (SELECT 1 FROM public.test_satellite_log + WHERE marker = 'native' AND value = 42 AND db_name = '_test_satellite_db' + AND role_name = 'df_e2e_user') THEN + RAISE EXCEPTION 'TEST FAILED: default target, captured vars, or execution role misrouted'; + END IF; + IF NOT EXISTS (SELECT 1 FROM df.nodes WHERE instance_id = inst_id AND node_type = 'SQL') OR + EXISTS (SELECT 1 FROM df.nodes AS node WHERE node.instance_id = inst_id + AND node.node_type = 'SQL' AND node.status IS DISTINCT FROM 'completed') THEN + RAISE EXCEPTION 'TEST FAILED: satellite node statuses not updated'; + END IF; + END $check$; +$remote$); + +-- Keep the returned ID outside the transaction being rolled back. +SELECT dblink_exec('satellite', 'BEGIN'); +CREATE TEMP TABLE _satellite_rolled_back AS +SELECT instance_id FROM dblink('satellite', $remote$ + SELECT df.start( + 'INSERT INTO public.test_satellite_log (marker) VALUES (''rolled-back'')', + 'test-satellite-rolled-back' + ) +$remote$) AS remote(instance_id TEXT); +SELECT dblink_exec('satellite', 'ROLLBACK'); + +SELECT dblink_exec('satellite', $remote$ + BEGIN; + INSERT INTO public.test_satellite_log (marker) VALUES ('caller-new'); +$remote$); +SELECT * FROM dblink('satellite', $remote$ + SELECT df.start( + 'INSERT INTO public.test_satellite_log (marker) VALUES (''independent'')', + 'test-satellite-independent', transaction_mode => 'new' + ) +$remote$) AS remote(instance_id TEXT); +SELECT dblink_exec('satellite', 'ROLLBACK'); + +SELECT dblink_exec('satellite', $remote$ + DO $check$ + DECLARE + inst_id TEXT := (SELECT id FROM df.instances WHERE label = 'test-satellite-independent'); + BEGIN + PERFORM df.await_instance(inst_id, 30); + IF inst_id IS NULL OR df.status(inst_id) IS DISTINCT FROM 'completed' THEN + RAISE EXCEPTION 'TEST FAILED: satellite independent start did not survive rollback'; + END IF; + IF EXISTS (SELECT 1 FROM public.test_satellite_log WHERE marker IN ('caller-new', 'rolled-back')) OR + NOT EXISTS (SELECT 1 FROM public.test_satellite_log WHERE marker = 'independent') OR + EXISTS (SELECT 1 FROM df.instances WHERE label = 'test-satellite-rolled-back') THEN + RAISE EXCEPTION 'TEST FAILED: satellite transaction boundary'; + END IF; + END $check$; +$remote$); + +SELECT dblink_exec('satellite', $remote$ + INSERT INTO _satellite_state SELECT 'children', df.start( + 'INSERT INTO public.test_satellite_log (marker) VALUES (''loop'')' + ~> ('INSERT INTO public.test_satellite_log (marker) VALUES (''left'')' + & 'INSERT INTO public.test_satellite_log (marker) VALUES (''right'')') + ~> df.loop( + 'UPDATE public.test_satellite_log SET value = value + 1 WHERE marker = ''loop''', + 'SELECT value < 2 FROM public.test_satellite_log WHERE marker = ''loop''', + continue_on_failure => true + ), 'test-satellite-children' + ); + INSERT INTO _satellite_state SELECT 'signal', df.start( + (df.wait_for_signal('go', 30) |=> 'payload') + ~> 'INSERT INTO public.test_satellite_log (marker, value) VALUES (''signal'', ($payload::jsonb->''data''->>''value'')::int)', + 'test-satellite-signal' + ); + INSERT INTO _satellite_state SELECT 'cancel', df.start( + df.wait_for_signal('never', 60) + ~> 'INSERT INTO public.test_satellite_log (marker) VALUES (''cancelled'')', + 'test-satellite-cancel' + ); + INSERT INTO _satellite_state SELECT 'http', df.start( + '{"node_type":"HTTP","query":"{\"url\":\"https://api.github.com/\",\"method\":\"GET\",\"body\":null,\"headers\":null,\"timeout_seconds\":5}"}', + 'test-satellite-http-denied' + ); +$remote$); + +SELECT dblink_exec('satellite', $remote$ + DO $check$ + DECLARE + inst_id TEXT := (SELECT instance_id FROM _satellite_state WHERE name = 'signal'); + status TEXT; + BEGIN + FOR attempt IN 1..200 LOOP + status := df.status(inst_id); + EXIT WHEN status IN ('completed', 'failed', 'cancelled'); + PERFORM df.signal(inst_id, 'go', '{"value":73}'); + PERFORM pg_sleep(0.1); + END LOOP; + PERFORM df.await_instance(inst_id, 10); + IF df.status(inst_id) IS DISTINCT FROM 'completed' OR + NOT EXISTS (SELECT 1 FROM public.test_satellite_log WHERE marker = 'signal' AND value = 73) THEN + RAISE EXCEPTION 'TEST FAILED: satellite signal not delivered'; + END IF; + END $check$; +$remote$); +SELECT * FROM dblink('satellite', $remote$ + SELECT df.cancel(instance_id, 'satellite test cancellation') + FROM _satellite_state WHERE name = 'cancel' +$remote$) AS remote(result TEXT); + +SELECT dblink_exec('satellite', $remote$ + DO $check$ + DECLARE + inst_id TEXT; + node_result TEXT; + BEGIN + SELECT instance_id INTO inst_id FROM _satellite_state WHERE name = 'cancel'; + IF df.await_instance(inst_id, 30) IS DISTINCT FROM 'cancelled' THEN + RAISE EXCEPTION 'TEST FAILED: satellite cancellation'; + END IF; + SELECT instance_id INTO inst_id FROM _satellite_state WHERE name = 'children'; + PERFORM df.await_instance(inst_id, 30); + IF df.status(inst_id) IS DISTINCT FROM 'completed' OR + (SELECT count(*) FROM public.test_satellite_log WHERE marker IN ('left', 'right')) <> 2 OR + (SELECT value FROM public.test_satellite_log WHERE marker = 'loop') IS DISTINCT FROM 2 THEN + RAISE EXCEPTION 'TEST FAILED: satellite parallel or loop children misrouted'; + END IF; + SELECT instance_id INTO inst_id FROM _satellite_state WHERE name = 'http'; + IF df.await_instance(inst_id, 30) IS DISTINCT FROM 'failed' THEN + RAISE EXCEPTION 'TEST FAILED: control HTTP permission authorized satellite request'; + END IF; + SELECT result::text INTO node_result FROM df.nodes + WHERE instance_id = inst_id AND node_type = 'HTTP'; + IF node_result IS NULL OR node_result NOT ILIKE '%does not have EXECUTE privilege%' THEN + RAISE EXCEPTION 'TEST FAILED: expected origin HTTP privilege denial, got %', node_result; + END IF; + IF EXISTS (SELECT 1 FROM public.test_satellite_log + WHERE marker IN ('rolled-back', 'cancelled') OR db_name <> '_test_satellite_db' + OR role_name <> 'df_e2e_user') THEN + RAISE EXCEPTION 'TEST FAILED: unexpected satellite side effect or execution identity'; + END IF; + END $check$; +$remote$); - IF err_msg NOT ILIKE '%must be created in database%' THEN - RAISE EXCEPTION 'TEST FAILED: Expected "must be created in database" in error, got: %', err_msg; +DO $$ +DECLARE + rollback_id TEXT := (SELECT instance_id FROM _satellite_rolled_back); + row_count INT; +BEGIN + SELECT total INTO row_count FROM dblink('satellite', format( + 'SELECT (SELECT count(*) FROM df.instances WHERE id = %L) + (SELECT count(*) FROM df.nodes WHERE instance_id = %L)', + rollback_id, rollback_id + )) AS remote(total INT); + IF row_count IS DISTINCT FROM 0 OR + EXISTS (SELECT 1 FROM df.instances WHERE label LIKE 'test-satellite-%') OR + EXISTS (SELECT 1 FROM df.vars WHERE name = 'satellite_value') OR + EXISTS (SELECT 1 FROM public.test_satellite_log) THEN + RAISE EXCEPTION 'TEST FAILED: satellite metadata or writes leaked into control'; END IF; - IF err_msg NOT ILIKE '%_test_wrong_db%' THEN - RAISE EXCEPTION 'TEST FAILED: Expected wrong db name in error, got: %', err_msg; + IF NOT EXISTS (SELECT 1 FROM pg_namespace WHERE nspname = df.duroxide_schema()) THEN + RAISE EXCEPTION 'TEST FAILED: control provider schema missing'; END IF; +END $$; - RAISE NOTICE 'PASSED: CREATE EXTENSION correctly rejected in wrong database'; - RAISE NOTICE 'Error was: %', err_msg; +SELECT dblink_exec('satellite', 'RESET SESSION AUTHORIZATION; DROP EXTENSION pg_durable CASCADE'); +SELECT dblink_disconnect('satellite'); +DROP DATABASE _test_satellite_db WITH (FORCE); + +SET SESSION AUTHORIZATION df_e2e_user; +CREATE TEMP TABLE _control_after_satellite AS +SELECT df.start('SELECT 84', 'test-control-after-satellite-drop') AS instance_id; +DO $$ +BEGIN + IF df.await_instance((SELECT instance_id FROM _control_after_satellite), 30) IS DISTINCT FROM 'completed' THEN + RAISE EXCEPTION 'TEST FAILED: dropping satellite stopped control execution'; + END IF; +END $$; +DROP TABLE _control_after_satellite; +RESET SESSION AUTHORIZATION; + +DO $$ +BEGIN + IF (SELECT epoch_id FROM df._worker_epoch) IS DISTINCT FROM (SELECT epoch_id FROM _control_epoch) THEN + RAISE EXCEPTION 'TEST FAILED: dropping satellite restarted control runtime'; + END IF; END $$; -DROP DATABASE IF EXISTS _test_wrong_db; +REVOKE EXECUTE ON FUNCTION df.http(text, text, text, jsonb, integer) FROM df_e2e_user; +REVOKE EXECUTE ON FUNCTION df.http_multipart(text, text, jsonb, jsonb, integer) FROM df_e2e_user; +DROP TABLE public.test_satellite_log; +DROP TABLE _control_epoch, _satellite_rolled_back; -- === Test: 34_multi_database === diff --git a/tests/e2e/sql/72_multi_database_lifecycle.sql b/tests/e2e/sql/72_multi_database_lifecycle.sql new file mode 100644 index 00000000..7de1d65a --- /dev/null +++ b/tests/e2e/sql/72_multi_database_lifecycle.sql @@ -0,0 +1,575 @@ +-- === Setup: owned databases; administration as postgres, workloads as df_e2e_user === + +CREATE EXTENSION IF NOT EXISTS dblink; + +DO $$ +BEGIN + IF current_database() IS DISTINCT FROM df.target_database() THEN + RAISE EXCEPTION 'TEST SETUP ERROR: run lifecycle tests in the control database'; + END IF; + IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'df_e2e_user' AND (rolsuper OR rolbypassrls)) THEN + RAISE EXCEPTION 'TEST SETUP ERROR: df_e2e_user must not bypass RLS'; + END IF; +END $$; + +BEGIN; +DELETE FROM df.nodes WHERE instance_id IN (SELECT id FROM df.instances WHERE label = 'e2e72-shadow'); +DELETE FROM df.instances WHERE label = 'e2e72-shadow'; +COMMIT; + +DROP DATABASE IF EXISTS _e2e72_origin WITH (FORCE); +DROP DATABASE IF EXISTS "_e2e72 satellite ""peer""" WITH (FORCE); +DROP DATABASE IF EXISTS _e2e72_target WITH (FORCE); +CREATE DATABASE _e2e72_origin; +CREATE DATABASE "_e2e72 satellite ""peer"""; +CREATE DATABASE _e2e72_target; + +CREATE TEMP TABLE _e2e72_databases (connection_name TEXT PRIMARY KEY, database_name TEXT NOT NULL); +INSERT INTO _e2e72_databases VALUES + ('e2e72_origin', '_e2e72_origin'), + ('e2e72_peer', '_e2e72 satellite "peer"'), + ('e2e72_target', '_e2e72_target'); +CREATE TEMP TABLE _e2e72_epoch AS SELECT epoch_id FROM df._worker_epoch; + +DROP TABLE IF EXISTS public.e2e72_log; +CREATE TABLE public.e2e72_log ( + marker TEXT PRIMARY KEY, + value TEXT NOT NULL, + database_name TEXT NOT NULL DEFAULT current_database(), + role_name TEXT NOT NULL DEFAULT current_user +); +GRANT SELECT, INSERT ON public.e2e72_log TO df_e2e_user; + +DO $$ +DECLARE + remote_db RECORD; +BEGIN + FOR remote_db IN SELECT * FROM _e2e72_databases ORDER BY connection_name LOOP + EXECUTE format('GRANT CONNECT ON DATABASE %I TO df_e2e_user', remote_db.database_name); + PERFORM dblink_connect(remote_db.connection_name, format( + 'host=localhost dbname=%L port=%s user=postgres', + remote_db.database_name, current_setting('port') + )); + IF remote_db.connection_name <> 'e2e72_target' THEN + PERFORM dblink_exec(remote_db.connection_name, 'CREATE EXTENSION pg_durable'); + PERFORM dblink_exec(remote_db.connection_name, + 'DO $grant$ BEGIN PERFORM df.grant_usage(''df_e2e_user''); END $grant$'); + END IF; + PERFORM dblink_exec(remote_db.connection_name, $remote$ + CREATE TABLE public.e2e72_log ( + marker TEXT PRIMARY KEY, + value TEXT NOT NULL, + database_name TEXT NOT NULL DEFAULT current_database(), + role_name TEXT NOT NULL DEFAULT current_user + ); + GRANT SELECT, INSERT ON public.e2e72_log TO df_e2e_user; + SET SESSION AUTHORIZATION df_e2e_user; + CREATE TEMP TABLE e2e72_state (scenario TEXT PRIMARY KEY, instance_id TEXT NOT NULL); + $remote$); + END LOOP; +END $$; + +-- === Explicit third target from a satellite; quoted satellite name on the SQLx route === + +SELECT dblink_exec('e2e72_peer', $remote$ + DO $setvar$ BEGIN PERFORM df.setvar('e2e72_value', 'satellite-captured'); END $setvar$; +$remote$); +SELECT dblink_exec('e2e72_peer', $remote$ + INSERT INTO e2e72_state SELECT 'third-target', df.start( + 'INSERT INTO public.e2e72_log (marker, value) VALUES (''third-first'', ''{e2e72_value}'')' + ~> 'INSERT INTO public.e2e72_log (marker, value) VALUES (''third-second'', current_database())', + 'e2e72-third-target', database => '_e2e72_target' + ); + INSERT INTO e2e72_state SELECT 'quoted-local', df.start( + 'INSERT INTO public.e2e72_log (marker, value) VALUES (''quoted-local'', current_database())', + 'e2e72-quoted-local' + ); +$remote$); +SELECT dblink_exec('e2e72_peer', $remote$ + DO $await$ + DECLARE + workflow RECORD; + final_status TEXT; + BEGIN + FOR workflow IN SELECT * FROM e2e72_state LOOP + final_status := df.await_instance(workflow.instance_id, 30); + IF final_status IS DISTINCT FROM 'completed' THEN + RAISE EXCEPTION 'TEST FAILED [routing %]: status = %', workflow.scenario, final_status; + END IF; + END LOOP; + END $await$; +$remote$); +SELECT dblink_exec('e2e72_peer', $remote$ + DO $check$ + DECLARE + target_id TEXT := (SELECT instance_id FROM e2e72_state WHERE scenario = 'third-target'); + local_id TEXT := (SELECT instance_id FROM e2e72_state WHERE scenario = 'quoted-local'); + BEGIN + IF NOT EXISTS (SELECT 1 FROM df.instances WHERE id = target_id AND database = '_e2e72_target') OR + NOT EXISTS (SELECT 1 FROM df.nodes WHERE instance_id = target_id) OR + EXISTS (SELECT 1 FROM df.nodes WHERE instance_id = target_id AND + (database IS DISTINCT FROM '_e2e72_target' OR status IS DISTINCT FROM 'completed')) THEN + RAISE EXCEPTION 'TEST FAILED [third target]: origin metadata missing or misrouted'; + END IF; + IF NOT EXISTS (SELECT 1 FROM df.instance_info(target_id) WHERE status = 'completed') OR + NOT EXISTS (SELECT 1 FROM df.instance_executions(target_id)) OR + NOT EXISTS (SELECT 1 FROM df.instance_info(local_id) WHERE status = 'completed') THEN + RAISE EXCEPTION 'TEST FAILED [quoted origin]: provider lookup failed'; + END IF; + IF (SELECT count(*) FROM public.e2e72_log) <> 1 OR NOT EXISTS ( + SELECT 1 FROM public.e2e72_log WHERE marker = 'quoted-local' + AND value = current_database() AND database_name = current_database() + AND role_name = 'df_e2e_user' + ) THEN + RAISE EXCEPTION 'TEST FAILED [quoted origin]: third-target writes leaked into origin'; + END IF; + END $check$; +$remote$); +SELECT dblink_exec('e2e72_target', $remote$ + DO $check$ + BEGIN + IF EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pg_durable') OR + (SELECT count(*) FROM public.e2e72_log) <> 2 OR + NOT EXISTS (SELECT 1 FROM public.e2e72_log + WHERE marker = 'third-first' AND value = 'satellite-captured') OR + NOT EXISTS (SELECT 1 FROM public.e2e72_log + WHERE marker = 'third-second' AND value = current_database()) OR + EXISTS (SELECT 1 FROM public.e2e72_log + WHERE database_name <> current_database() OR role_name <> 'df_e2e_user') THEN + RAISE EXCEPTION 'TEST FAILED [third target]: wrong writes, variables, or execution identity'; + END IF; + END $check$; +$remote$); + +DO $$ +BEGIN + IF EXISTS (SELECT 1 FROM public.e2e72_log) OR + EXISTS (SELECT 1 FROM df.instances WHERE label IN ('e2e72-third-target', 'e2e72-quoted-local')) OR + EXISTS (SELECT 1 FROM df.vars WHERE name = 'e2e72_value') THEN + RAISE EXCEPTION 'TEST FAILED [control isolation]: satellite state leaked into control'; + END IF; +END $$; + +-- === Same local ID in two satellites and control; metadata-only shadows, no engine forgery === + +SELECT dblink_exec('e2e72_origin', $remote$ + INSERT INTO e2e72_state SELECT 'collision', df.start( + (df.wait_for_signal('e2e72-release', 120) |=> 'payload') + ~> 'INSERT INTO public.e2e72_log (marker, value) VALUES (''released'', $payload::jsonb->''data''->>''source'')', + 'e2e72-collision-owner' + ); +$remote$); +SELECT dblink_exec('e2e72_origin', $remote$ + DO $wait$ + DECLARE + local_id TEXT := (SELECT instance_id FROM e2e72_state WHERE scenario = 'collision'); + waiting BOOLEAN := false; + BEGIN + FOR attempt IN 1..300 LOOP + SELECT EXISTS (SELECT 1 FROM df.nodes + WHERE instance_id = local_id AND node_type = 'SIGNAL' AND status = 'running') INTO waiting; + EXIT WHEN waiting; + PERFORM pg_sleep(0.1); + END LOOP; + IF NOT waiting OR df.status(local_id) IS DISTINCT FROM 'running' THEN + RAISE EXCEPTION 'TEST FAILED [collision setup]: owner did not reach signal wait'; + END IF; + END $wait$; +$remote$); + +CREATE TEMP TABLE _e2e72_collision AS +SELECT instance_id FROM dblink('e2e72_origin', + 'SELECT instance_id FROM e2e72_state WHERE scenario = ''collision''') AS remote(instance_id TEXT); +GRANT SELECT ON _e2e72_collision TO df_e2e_user; + +DO $$ +DECLARE + local_id TEXT := (SELECT instance_id FROM _e2e72_collision); +BEGIN + IF local_id IS NULL OR local_id !~ '^[0-9a-f]{8}$' THEN + RAISE EXCEPTION 'TEST FAILED [collision setup]: missing public local ID'; + END IF; + INSERT INTO df.instances (id, root_node, submitted_by, status, label) + VALUES (local_id, '00000072', 'df_e2e_user'::regrole, 'completed', 'e2e72-shadow'); + INSERT INTO df.nodes (id, instance_id, node_type, query, submitted_by, status, result) + VALUES ('00000072', local_id, 'SQL', 'SELECT ''control-shadow''', + 'df_e2e_user'::regrole, 'completed', '"control-shadow"'::jsonb); + PERFORM dblink_exec('e2e72_peer', 'RESET SESSION AUTHORIZATION'); + PERFORM dblink_exec('e2e72_peer', format($remote$ + BEGIN; + INSERT INTO e2e72_state VALUES ('shadow', %1$L); + INSERT INTO df.instances (id, root_node, submitted_by, status, label) + VALUES (%1$L, '00000072', 'postgres'::regrole, 'completed', 'e2e72-shadow'); + INSERT INTO df.nodes (id, instance_id, node_type, query, submitted_by, status, result) + VALUES ('00000072', %1$L, 'SQL', 'SELECT ''peer-shadow''', + 'postgres'::regrole, 'completed', '"peer-shadow"'::jsonb); + COMMIT; + SET SESSION AUTHORIZATION df_e2e_user; + $remote$, local_id)); +END $$; + +SELECT dblink_exec('e2e72_peer', $remote$ + DO $check$ + DECLARE + local_id TEXT := (SELECT instance_id FROM e2e72_state WHERE scenario = 'shadow'); + denied BOOLEAN := false; + BEGIN + IF EXISTS (SELECT 1 FROM df.instances WHERE id = local_id) OR + EXISTS (SELECT 1 FROM df.nodes WHERE instance_id = local_id) OR + EXISTS (SELECT 1 FROM df.instance_info(local_id)) OR + EXISTS (SELECT 1 FROM df.instance_executions(local_id)) OR + EXISTS (SELECT 1 FROM df.list_instances() WHERE instance_id = local_id) OR + df.status(local_id) IS NOT NULL OR df.result(local_id) IS NOT NULL THEN + RAISE EXCEPTION 'TEST FAILED [satellite RLS]: unowned metadata or provider history leaked'; + END IF; + BEGIN + PERFORM df.signal(local_id, 'e2e72-release', '{"source":"rls-denied"}'); + EXCEPTION WHEN OTHERS THEN + IF SQLERRM IS DISTINCT FROM 'Instance not found or access denied: ' || local_id THEN + RAISE; + END IF; + denied := true; + END; + IF NOT denied THEN + RAISE EXCEPTION 'TEST FAILED [satellite RLS]: unowned signal accepted'; + END IF; + denied := false; + BEGIN + PERFORM df.cancel(local_id, 'e2e72 RLS probe'); + EXCEPTION WHEN OTHERS THEN + IF SQLERRM IS DISTINCT FROM 'Instance not found or access denied: ' || local_id THEN + RAISE; + END IF; + denied := true; + END; + IF NOT denied THEN + RAISE EXCEPTION 'TEST FAILED [satellite RLS]: unowned cancellation accepted'; + END IF; + END $check$; +$remote$); + +SELECT dblink_exec('e2e72_peer', $remote$ + RESET SESSION AUTHORIZATION; + BEGIN; + UPDATE df.instances SET submitted_by = 'df_e2e_user'::regrole WHERE label = 'e2e72-shadow'; + UPDATE df.nodes SET submitted_by = 'df_e2e_user'::regrole + WHERE instance_id = (SELECT instance_id FROM e2e72_state WHERE scenario = 'shadow'); + COMMIT; + SET SESSION AUTHORIZATION df_e2e_user; +$remote$); +SELECT dblink_exec('e2e72_peer', $remote$ + DO $check$ + DECLARE + local_id TEXT := (SELECT instance_id FROM e2e72_state WHERE scenario = 'shadow'); + cancel_result TEXT; + BEGIN + IF df.status(local_id) IS DISTINCT FROM 'completed' OR + df.result(local_id)::jsonb IS DISTINCT FROM '"peer-shadow"'::jsonb THEN + RAISE EXCEPTION 'TEST FAILED [peer shadow]: local metadata/result did not stay local'; + END IF; + IF EXISTS (SELECT 1 FROM df.instance_info(local_id)) OR + EXISTS (SELECT 1 FROM df.instance_executions(local_id)) THEN + RAISE EXCEPTION 'TEST FAILED [peer shadow]: resolved another installation engine'; + END IF; + PERFORM df.signal(local_id, 'e2e72-release', '{"source":"wrong-peer"}'); + cancel_result := df.cancel(local_id, 'e2e72 peer shadow cancellation'); + IF cancel_result IS DISTINCT FROM format('Instance %s cancelled: e2e72 peer shadow cancellation', local_id) THEN + RAISE EXCEPTION 'TEST FAILED [peer shadow cancellation]: %', cancel_result; + END IF; + END $check$; +$remote$); + +SET SESSION AUTHORIZATION df_e2e_user; +DO $$ +DECLARE + local_id TEXT := (SELECT instance_id FROM _e2e72_collision); + cancel_result TEXT; +BEGIN + IF df.status(local_id) IS DISTINCT FROM 'completed' OR + df.result(local_id)::jsonb IS DISTINCT FROM '"control-shadow"'::jsonb THEN + RAISE EXCEPTION 'TEST FAILED [control shadow]: local metadata/result did not stay local'; + END IF; + IF EXISTS (SELECT 1 FROM df.instance_info(local_id)) OR + EXISTS (SELECT 1 FROM df.instance_executions(local_id)) THEN + RAISE EXCEPTION 'TEST FAILED [control shadow]: resolved a satellite engine'; + END IF; + PERFORM df.signal(local_id, 'e2e72-release', '{"source":"wrong-control"}'); + cancel_result := df.cancel(local_id, 'e2e72 control shadow cancellation'); + IF cancel_result IS DISTINCT FROM format('Instance %s cancelled: e2e72 control shadow cancellation', local_id) THEN + RAISE EXCEPTION 'TEST FAILED [control shadow cancellation]: %', cancel_result; + END IF; +END $$; +RESET SESSION AUTHORIZATION; + +SELECT dblink_exec('e2e72_origin', $remote$ + DO $release$ + DECLARE + local_id TEXT := (SELECT instance_id FROM e2e72_state WHERE scenario = 'collision'); + BEGIN + IF df.status(local_id) IS DISTINCT FROM 'running' OR + NOT EXISTS (SELECT 1 FROM df.instance_info(local_id) WHERE status = 'running') THEN + RAISE EXCEPTION 'TEST FAILED [owner isolation]: foreign shadow disturbed live owner'; + END IF; + FOR attempt IN 1..300 LOOP + EXIT WHEN df.status(local_id) IN ('completed', 'failed', 'cancelled'); + PERFORM df.signal(local_id, 'e2e72-release', '{"source":"owner"}'); + PERFORM pg_sleep(0.1); + END LOOP; + IF df.await_instance(local_id, 10) IS DISTINCT FROM 'completed' THEN + RAISE EXCEPTION 'TEST FAILED [owner isolation]: foreign cancellation reached owner'; + END IF; + END $release$; +$remote$); +SELECT dblink_exec('e2e72_origin', $remote$ + DO $check$ + DECLARE + local_id TEXT := (SELECT instance_id FROM e2e72_state WHERE scenario = 'collision'); + BEGIN + IF (SELECT count(*) FROM public.e2e72_log) <> 1 OR NOT EXISTS ( + SELECT 1 FROM public.e2e72_log WHERE marker = 'released' AND value = 'owner' + AND database_name = current_database() AND role_name = 'df_e2e_user' + ) OR NOT EXISTS (SELECT 1 FROM df.instance_info(local_id) WHERE status = 'completed') OR + NOT EXISTS (SELECT 1 FROM df.instance_executions(local_id)) THEN + RAISE EXCEPTION 'TEST FAILED [owner isolation]: wrong signal payload, effects, or engine history'; + END IF; + END $check$; +$remote$); + +BEGIN; +DELETE FROM df.nodes WHERE instance_id IN (SELECT instance_id FROM _e2e72_collision); +DELETE FROM df.instances WHERE id IN (SELECT instance_id FROM _e2e72_collision) AND label = 'e2e72-shadow'; +COMMIT; +SELECT dblink_exec('e2e72_peer', $remote$ + RESET SESSION AUTHORIZATION; + BEGIN; + DELETE FROM df.nodes WHERE instance_id = (SELECT instance_id FROM e2e72_state WHERE scenario = 'shadow'); + DELETE FROM df.instances WHERE label = 'e2e72-shadow'; + COMMIT; + SET SESSION AUTHORIZATION df_e2e_user; +$remote$); + +-- === Reinstall during a persisted timer: cached old graph must not reach the replacement === + +SELECT dblink_exec('e2e72_origin', $remote$ + INSERT INTO e2e72_state SELECT 'old-timer', df.start( + 'INSERT INTO public.e2e72_log (marker, value) VALUES (''armed'', ''old-installation'')' + ~> df.sleep(10) + ~> 'INSERT INTO public.e2e72_log (marker, value) VALUES (''stale-write'', ''old-installation'')', + 'e2e72-old-timer' + ); +$remote$); +SELECT dblink_exec('e2e72_origin', $remote$ + DO $wait$ + DECLARE + local_id TEXT := (SELECT instance_id FROM e2e72_state WHERE scenario = 'old-timer'); + waiting BOOLEAN := false; + BEGIN + FOR attempt IN 1..300 LOOP + SELECT EXISTS (SELECT 1 FROM df.nodes + WHERE instance_id = local_id AND node_type = 'SLEEP' AND status = 'running') INTO waiting; + EXIT WHEN waiting; + PERFORM pg_sleep(0.1); + END LOOP; + IF NOT waiting OR df.status(local_id) IS DISTINCT FROM 'running' THEN + RAISE EXCEPTION 'TEST FAILED [reinstall setup]: old graph did not reach its timer'; + END IF; + END $wait$; +$remote$); +SELECT dblink_exec('e2e72_origin', $remote$ + DO $check$ + BEGIN + IF NOT EXISTS (SELECT 1 FROM public.e2e72_log WHERE marker = 'armed' AND value = 'old-installation') OR + EXISTS (SELECT 1 FROM public.e2e72_log WHERE marker = 'stale-write') THEN + RAISE EXCEPTION 'TEST FAILED [reinstall setup]: old graph not armed before replacement'; + END IF; + END $check$; + CREATE TEMP TABLE e2e72_old AS + SELECT to_jsonb(instance) AS instance_row, + (SELECT jsonb_agg(to_jsonb(node) ORDER BY node.id) FROM df.nodes AS node + WHERE node.instance_id = instance.id) AS node_rows, + (SELECT id FROM df._installation WHERE singleton) AS installation_id, + (SELECT oid FROM pg_database WHERE datname = current_database()) AS database_oid + FROM df.instances AS instance + WHERE id = (SELECT instance_id FROM e2e72_state WHERE scenario = 'old-timer'); +$remote$); + +CREATE TEMP TABLE _e2e72_old AS +SELECT *, NULL::TEXT AS engine_id FROM dblink('e2e72_origin', + 'SELECT instance_row->>''id'', installation_id, database_oid FROM e2e72_old' +) AS remote(local_id TEXT, installation_id UUID, database_oid OID); + +DO $$ +DECLARE + old_identity RECORD; + provider_schema TEXT := df.duroxide_schema(); + actual_engine_id TEXT; + timer_waiting BOOLEAN := false; +BEGIN + SELECT * INTO STRICT old_identity FROM _e2e72_old; + FOR attempt IN 1..300 LOOP + EXECUTE format('SELECT instance_id FROM %I.instances WHERE instance_id = $1', provider_schema) + INTO actual_engine_id USING format('pgdf-%s-%s-%s', old_identity.database_oid, + replace(old_identity.installation_id::text, '-', ''), old_identity.local_id); + EXECUTE format( + 'SELECT EXISTS (SELECT 1 FROM %1$I.history WHERE instance_id = $1 + AND event_data::jsonb->>''type'' = ''TimerCreated'') + AND NOT EXISTS (SELECT 1 FROM %1$I.history WHERE instance_id = $1 + AND event_data::jsonb->>''type'' = ''TimerFired'')', provider_schema + ) INTO timer_waiting USING actual_engine_id; + EXIT WHEN actual_engine_id IS NOT NULL AND timer_waiting; + PERFORM pg_sleep(0.1); + END LOOP; + IF actual_engine_id IS NULL OR NOT timer_waiting THEN + RAISE EXCEPTION 'TEST FAILED [reinstall setup]: no persisted unfired timer for old engine'; + END IF; + UPDATE _e2e72_old SET engine_id = actual_engine_id; +END $$; + +SELECT dblink_exec('e2e72_origin', $remote$ + RESET SESSION AUTHORIZATION; + BEGIN; + DROP EXTENSION pg_durable CASCADE; + DROP TABLE public.e2e72_log; + CREATE EXTENSION pg_durable; + DO $grant$ BEGIN PERFORM df.grant_usage('df_e2e_user'); END $grant$; + CREATE TABLE public.e2e72_log ( + marker TEXT PRIMARY KEY, + value TEXT NOT NULL, + database_name TEXT NOT NULL DEFAULT current_database(), + role_name TEXT NOT NULL DEFAULT current_user + ); + GRANT SELECT, INSERT ON public.e2e72_log TO df_e2e_user; + DO $identity$ + BEGIN + IF (SELECT oid FROM pg_database WHERE datname = current_database()) IS DISTINCT FROM + (SELECT database_oid FROM e2e72_old) OR + (SELECT id FROM df._installation WHERE singleton) IS NOT DISTINCT FROM + (SELECT installation_id FROM e2e72_old) THEN + RAISE EXCEPTION 'TEST FAILED [reinstall identity]: expected same database OID and new installation UUID'; + END IF; + END $identity$; + INSERT INTO df.instances + SELECT (jsonb_populate_record(NULL::df.instances, instance_row || jsonb_build_object( + 'status', 'completed', 'label', 'e2e72-replacement-shadow', + 'created_at', now(), 'updated_at', now(), 'completed_at', now() + ))).* FROM e2e72_old; + INSERT INTO df.nodes + SELECT (jsonb_populate_record(NULL::df.nodes, saved.node_row || jsonb_build_object( + 'status', 'completed', 'result', 'replacement-sentinel', 'error', NULL, + 'status_details', NULL, 'created_at', now(), 'updated_at', now() + ))).* FROM e2e72_old CROSS JOIN LATERAL jsonb_array_elements(node_rows) AS saved(node_row); + CREATE TEMP TABLE e2e72_replacement_snapshot AS + SELECT to_jsonb(instance) AS instance_row, + (SELECT jsonb_agg(to_jsonb(node) ORDER BY node.id) FROM df.nodes AS node + WHERE node.instance_id = instance.id) AS node_rows + FROM df.instances AS instance WHERE label = 'e2e72-replacement-shadow'; + GRANT SELECT ON e2e72_replacement_snapshot TO df_e2e_user; + COMMIT; + SET SESSION AUTHORIZATION df_e2e_user; +$remote$); + +SELECT dblink_exec('e2e72_origin', $remote$ + INSERT INTO e2e72_state SELECT 'replacement-new', df.start( + 'INSERT INTO public.e2e72_log (marker, value) VALUES (''replacement-new'', ''new-installation'')', + 'e2e72-replacement-new' + ); +$remote$); +SET SESSION AUTHORIZATION df_e2e_user; +CREATE TEMP TABLE _e2e72_control AS SELECT df.start( + 'INSERT INTO public.e2e72_log (marker, value) VALUES (''control-new'', ''control'')', + 'e2e72-control-after-reinstall' +) AS instance_id; +DO $$ +BEGIN + IF df.await_instance((SELECT instance_id FROM _e2e72_control), 30) IS DISTINCT FROM 'completed' THEN + RAISE EXCEPTION 'TEST FAILED [reinstall control]: control work did not complete'; + END IF; +END $$; +RESET SESSION AUTHORIZATION; +SELECT dblink_exec('e2e72_origin', $remote$ + DO $await$ + BEGIN + IF df.await_instance((SELECT instance_id FROM e2e72_state WHERE scenario = 'replacement-new'), 30) + IS DISTINCT FROM 'completed' THEN + RAISE EXCEPTION 'TEST FAILED [reinstall new work]: replacement work did not complete'; + END IF; + END $await$; +$remote$); + +DO $$ +DECLARE + old_engine_id TEXT := (SELECT engine_id FROM _e2e72_old); + provider_schema TEXT := df.duroxide_schema(); + engine_status TEXT; + engine_output TEXT; + timer_fired BOOLEAN; +BEGIN + FOR attempt IN 1..1300 LOOP + EXECUTE format('SELECT status, output FROM %I.get_instance_info($1)', provider_schema) + INTO engine_status, engine_output USING old_engine_id; + EXIT WHEN lower(engine_status) IN ('completed', 'failed', 'cancelled'); + PERFORM pg_sleep(0.1); + END LOOP; + IF lower(engine_status) IS DISTINCT FROM 'failed' OR + position('Origin installation removed or replaced' IN coalesce(engine_output, '')) = 0 THEN + RAISE EXCEPTION 'TEST FAILED [old engine]: expected replaced-installation failure for %, status=%, output=%', + old_engine_id, engine_status, engine_output; + END IF; + EXECUTE format('SELECT EXISTS (SELECT 1 FROM %I.history WHERE instance_id = $1 + AND event_data::jsonb->>''type'' = ''TimerFired'')', provider_schema) + INTO timer_fired USING old_engine_id; + IF NOT timer_fired THEN + RAISE EXCEPTION 'TEST FAILED [old engine]: old cached graph never resumed after its timer'; + END IF; +END $$; + +SELECT dblink_exec('e2e72_origin', $remote$ + DO $check$ + DECLARE + old_id TEXT := (SELECT instance_row->>'id' FROM e2e72_old); + new_id TEXT := (SELECT instance_id FROM e2e72_state WHERE scenario = 'replacement-new'); + BEGIN + IF (SELECT count(*) FROM public.e2e72_log) <> 1 OR NOT EXISTS ( + SELECT 1 FROM public.e2e72_log WHERE marker = 'replacement-new' AND value = 'new-installation' + AND database_name = current_database() AND role_name = 'df_e2e_user' + ) THEN + RAISE EXCEPTION 'TEST FAILED [replacement effects]: old graph touched the replacement table'; + END IF; + IF (SELECT to_jsonb(instance) FROM df.instances AS instance WHERE id = old_id) IS DISTINCT FROM + (SELECT instance_row FROM e2e72_replacement_snapshot) OR + (SELECT jsonb_agg(to_jsonb(node) ORDER BY node.id) FROM df.nodes AS node WHERE instance_id = old_id) + IS DISTINCT FROM (SELECT node_rows FROM e2e72_replacement_snapshot) THEN + RAISE EXCEPTION 'TEST FAILED [replacement metadata]: old engine mutated new installation rows'; + END IF; + IF EXISTS (SELECT 1 FROM df.instance_info(old_id)) OR + EXISTS (SELECT 1 FROM df.instance_executions(old_id)) OR + NOT EXISTS (SELECT 1 FROM df.instance_info(new_id) WHERE status = 'completed') THEN + RAISE EXCEPTION 'TEST FAILED [replacement namespace]: lookup crossed installation UUIDs'; + END IF; + END $check$; +$remote$); + +DO $$ +BEGIN + IF (SELECT count(*) FROM public.e2e72_log) <> 1 OR NOT EXISTS ( + SELECT 1 FROM public.e2e72_log WHERE marker = 'control-new' AND value = 'control' + AND database_name = current_database() AND role_name = 'df_e2e_user' + ) THEN + RAISE EXCEPTION 'TEST FAILED [control effects]: satellite work touched control'; + END IF; + IF (SELECT count(*) FROM _e2e72_epoch) <> 1 OR + (SELECT epoch_id FROM df._worker_epoch) IS DISTINCT FROM (SELECT epoch_id FROM _e2e72_epoch) THEN + RAISE EXCEPTION 'TEST FAILED [control epoch]: satellite reinstall restarted the control runtime'; + END IF; +END $$; + +-- === Cleanup: only this file's fixtures === + +SELECT dblink_disconnect(connection_name) FROM _e2e72_databases; +DROP DATABASE _e2e72_origin WITH (FORCE); +DROP DATABASE "_e2e72 satellite ""peer""" WITH (FORCE); +DROP DATABASE _e2e72_target WITH (FORCE); +DROP TABLE public.e2e72_log; +DROP TABLE _e2e72_databases, _e2e72_epoch, _e2e72_collision, _e2e72_old, _e2e72_control; + +SELECT 'TEST PASSED: multi-database lifecycle' AS result; \ No newline at end of file diff --git a/tests/e2e/sql/73_multi_database_reconcile.sql b/tests/e2e/sql/73_multi_database_reconcile.sql new file mode 100644 index 00000000..9e004812 --- /dev/null +++ b/tests/e2e/sql/73_multi_database_reconcile.sql @@ -0,0 +1,420 @@ +CREATE EXTENSION IF NOT EXISTS dblink; + +DO $$ +BEGIN + IF current_database() IS DISTINCT FROM df.target_database() OR + current_setting('pg_durable.retention_days') <> '0' OR + current_setting('pg_durable.reconcile_interval') <> '2' THEN + RAISE EXCEPTION 'TEST SETUP ERROR: use the reconcile phase'; + END IF; + IF (SELECT atttypid FROM pg_attribute + WHERE attrelid = format('%I.instances', df.duroxide_schema())::regclass + AND attname = 'created_at') IS DISTINCT FROM 'timestamptz'::regtype::oid OR + (SELECT atttypid FROM pg_attribute + WHERE attrelid = format('%I.executions', df.duroxide_schema())::regclass + AND attname = 'completed_at') IS DISTINCT FROM 'timestamptz'::regtype::oid OR + (SELECT atttypid FROM pg_attribute + WHERE attrelid = format('%I.executions', df.duroxide_schema())::regclass + AND attname = 'status') IS DISTINCT FROM 'text'::regtype::oid THEN + RAISE EXCEPTION 'TEST FAILED [provider types]: expected timestamptz timestamps and text status'; + END IF; +END $$; + +DROP DATABASE IF EXISTS _e2e73_origin WITH (FORCE); +DROP DATABASE IF EXISTS _e2e73_peer WITH (FORCE); +DROP DATABASE IF EXISTS _e2e73_removed WITH (FORCE); +CREATE DATABASE _e2e73_origin; +CREATE DATABASE _e2e73_peer; +CREATE DATABASE _e2e73_removed; + +CREATE TEMP TABLE _e2e73_ids ( + origin TEXT, + scenario TEXT, + local_id TEXT NOT NULL, + engine_id TEXT NOT NULL, + PRIMARY KEY (origin, scenario) +); + +DO $$ +DECLARE + origin_name TEXT; +BEGIN + FOREACH origin_name IN ARRAY ARRAY['_e2e73_origin', '_e2e73_peer', '_e2e73_removed'] LOOP + PERFORM dblink_connect(origin_name, format( + 'host=localhost port=%s dbname=%L user=postgres', current_setting('port'), origin_name)); + PERFORM dblink_exec(origin_name, $remote$ + CREATE EXTENSION pg_durable; + CREATE TEMP TABLE test_state (scenario TEXT PRIMARY KEY, local_id TEXT); + CREATE FUNCTION public.e2e73_hold_metadata() RETURNS trigger LANGUAGE plpgsql AS $hold$ + BEGIN + IF NEW.label LIKE 'e2e73-held-%' THEN + NEW.created_at := clock_timestamp() + interval '31 days'; + IF NEW.status IN ('completed', 'failed', 'cancelled') THEN + NEW.completed_at := clock_timestamp() + interval '31 days'; + END IF; + END IF; + RETURN NEW; + END $hold$; + CREATE TRIGGER e2e73_hold_metadata BEFORE INSERT OR UPDATE ON df.instances + FOR EACH ROW EXECUTE FUNCTION public.e2e73_hold_metadata(); + $remote$); + END LOOP; +END $$; + +CREATE FUNCTION pg_temp.capture_ids(origin_name TEXT) RETURNS void LANGUAGE plpgsql AS $$ +BEGIN + INSERT INTO _e2e73_ids + SELECT origin_name, remote.scenario, remote.local_id, remote.engine_id + FROM dblink(origin_name, $remote$ + SELECT scenario, local_id, + 'pgdf-' || (SELECT oid::text FROM pg_database WHERE datname = current_database()) || + '-' || (SELECT replace(id::text, '-', '') FROM df._installation) || '-' || local_id + FROM test_state + $remote$) AS remote(scenario TEXT, local_id TEXT, engine_id TEXT) + ON CONFLICT (origin, scenario) DO NOTHING; +END $$; + +CREATE FUNCTION pg_temp.engine_status(engine_id TEXT) RETURNS TEXT LANGUAGE plpgsql AS $$ +DECLARE + current_status TEXT; +BEGIN + EXECUTE format('SELECT e.status FROM %1$I.instances i JOIN %1$I.executions e + ON e.instance_id = i.instance_id AND e.execution_id = i.current_execution_id + WHERE i.instance_id = $1', df.duroxide_schema()) INTO current_status USING engine_id; + RETURN current_status; +END $$; + +CREATE FUNCTION pg_temp.wait_engine(engine_id TEXT, expected TEXT) RETURNS void LANGUAGE plpgsql AS $$ +BEGIN + FOR attempt IN 1..600 LOOP + IF pg_temp.engine_status(engine_id) IS NOT DISTINCT FROM expected THEN + RETURN; + END IF; + PERFORM pg_sleep(0.1); + END LOOP; + RAISE EXCEPTION 'TEST FAILED [engine wait]: % expected %, got %', + engine_id, expected, pg_temp.engine_status(engine_id); +END $$; + +CREATE FUNCTION pg_temp.wait_subscription(engine_id TEXT, signal_name TEXT) RETURNS void LANGUAGE plpgsql AS $$ +DECLARE + subscribed BOOLEAN; +BEGIN + FOR attempt IN 1..300 LOOP + EXECUTE format('SELECT EXISTS (SELECT 1 FROM %1$I.history h JOIN %1$I.instances i + ON i.instance_id = h.instance_id AND i.current_execution_id = h.execution_id + WHERE h.instance_id = $1 AND h.event_data::jsonb->>''type'' = ''ExternalSubscribed'' + AND h.event_data::jsonb->>''name'' = $2)', df.duroxide_schema()) + INTO subscribed USING engine_id, signal_name; + IF subscribed THEN + RETURN; + END IF; + PERFORM pg_sleep(0.1); + END LOOP; + RAISE EXCEPTION 'TEST FAILED [subscription wait]: % never subscribed to % (engine status: %)', + engine_id, signal_name, pg_temp.engine_status(engine_id); +END $$; + +CREATE FUNCTION pg_temp.assert_engine_gone(engine_id TEXT) RETURNS void LANGUAGE plpgsql AS $$ +DECLARE + table_name TEXT; + remaining BIGINT; +BEGIN + FOREACH table_name IN ARRAY ARRAY['instances', 'executions', 'history', + 'orchestrator_queue', 'worker_queue', 'instance_locks'] LOOP + EXECUTE format('SELECT count(*) FROM %I.%I + WHERE instance_id = $1 OR instance_id LIKE $1 || ''::%%''', + df.duroxide_schema(), table_name) INTO remaining USING engine_id; + IF remaining <> 0 THEN + RAISE EXCEPTION 'TEST FAILED [engine cleanup]: % retains % rows for %', table_name, remaining, engine_id; + END IF; + END LOOP; +END $$; + +SELECT dblink_exec('_e2e73_origin', $remote$ + INSERT INTO test_state VALUES ('completed', df.start('SELECT 73', 'e2e73-held-completed')); + INSERT INTO test_state VALUES ('failed', df.start('SELECT 1 / 0', 'e2e73-held-failed')); + INSERT INTO test_state VALUES ('cancelled', df.start(df.wait_for_signal('cancel-me'), 'e2e73-held-cancelled')); + INSERT INTO test_state VALUES ('live', df.start(df.wait_for_signal('release'), 'e2e73-live')); +$remote$); +SELECT dblink_exec('_e2e73_peer', $remote$ + CREATE TABLE public._e2e73_success (marker TEXT PRIMARY KEY); + INSERT INTO test_state VALUES ('failed', df.start('SELECT 1 / 0', 'e2e73-held-peer')); + INSERT INTO test_state VALUES ('live', df.start(df.wait_for_signal('release') + ~> 'INSERT INTO public._e2e73_success VALUES (''peer'') ON CONFLICT DO NOTHING', 'e2e73-peer-live')); +$remote$); +SELECT dblink_exec('_e2e73_removed', $remote$ + INSERT INTO test_state VALUES ('removed', df.start( + 'SELECT 73' ~> df.wait_for_signal('never-release'), 'e2e73-removed')); +$remote$); +SELECT pg_temp.capture_ids(origin_name) +FROM unnest(ARRAY['_e2e73_origin', '_e2e73_peer', '_e2e73_removed']) AS origins(origin_name); + +SELECT pg_temp.wait_engine(engine_id, CASE WHEN scenario = 'completed' THEN 'Completed' + WHEN scenario = 'failed' THEN 'Failed' ELSE 'Running' END) FROM _e2e73_ids; +SELECT dblink_exec('_e2e73_origin', $remote$ + DO $cancel$ BEGIN + PERFORM df.cancel((SELECT local_id FROM test_state WHERE scenario = 'cancelled'), 'e2e73 cancellation'); + END $cancel$; +$remote$); +SELECT pg_temp.wait_engine(engine_id, 'Failed') FROM _e2e73_ids WHERE scenario = 'cancelled'; + +DO $$ +DECLARE + origin_name TEXT; +BEGIN + FOREACH origin_name IN ARRAY ARRAY['_e2e73_origin', '_e2e73_peer', '_e2e73_removed'] LOOP + PERFORM dblink_exec(origin_name, $remote$ + DO $waiting$ + BEGIN + FOR attempt IN 1..300 LOOP + EXIT WHEN EXISTS (SELECT 1 FROM df.nodes WHERE node_type = 'SIGNAL' AND status = 'running'); + PERFORM pg_sleep(0.1); + END LOOP; + IF NOT EXISTS (SELECT 1 FROM df.nodes WHERE node_type = 'SIGNAL' AND status = 'running') THEN + RAISE EXCEPTION 'TEST FAILED [setup]: no waiting signal'; + END IF; + IF EXISTS (SELECT 1 FROM test_state s LEFT JOIN df.instances i ON i.id = s.local_id + WHERE s.scenario IN ('completed', 'failed', 'cancelled') AND + (i.status IS DISTINCT FROM s.scenario OR NOT EXISTS + (SELECT 1 FROM df.nodes n WHERE n.instance_id = s.local_id))) THEN + RAISE EXCEPTION 'TEST FAILED [terminal metadata]: missing or wrong terminal rows'; + END IF; + END $waiting$; + $remote$); + END LOOP; +END $$; + +-- A local-ID collision must not let satellite retention remove control metadata. +BEGIN; +INSERT INTO df.instances (id, root_node, submitted_by, status, label) +SELECT local_id, '00000073', 'postgres'::regrole, 'running', 'e2e73-shadow' +FROM _e2e73_ids WHERE origin = '_e2e73_origin' AND scenario = 'completed'; +INSERT INTO df.nodes (id, instance_id, node_type, query, submitted_by, status) +SELECT '00000073', local_id, 'SQL', 'SELECT 73', 'postgres'::regrole, 'running' +FROM _e2e73_ids WHERE origin = '_e2e73_origin' AND scenario = 'completed'; +COMMIT; + +SELECT dblink_exec('_e2e73_origin', $remote$ + DROP TRIGGER e2e73_hold_metadata ON df.instances; + UPDATE df.instances SET created_at = clock_timestamp() - interval '1 minute', + completed_at = clock_timestamp() - interval '1 minute' + WHERE label LIKE 'e2e73-held-%'; +$remote$); +SELECT pg_temp.wait_engine(engine_id, NULL) FROM _e2e73_ids +WHERE origin = '_e2e73_origin' AND scenario IN ('completed', 'failed', 'cancelled'); +SELECT dblink_exec('_e2e73_origin', $remote$ + DO $clean$ + BEGIN + FOR attempt IN 1..300 LOOP + EXIT WHEN NOT EXISTS (SELECT 1 FROM df.instances WHERE label LIKE 'e2e73-held-%'); + PERFORM pg_sleep(0.1); + END LOOP; + IF EXISTS (SELECT 1 FROM df.instances WHERE label LIKE 'e2e73-held-%') OR + EXISTS (SELECT 1 FROM df.nodes WHERE instance_id IN + (SELECT local_id FROM test_state WHERE scenario IN ('completed', 'failed', 'cancelled'))) THEN + RAISE EXCEPTION 'TEST FAILED [terminal retention]: satellite metadata survived'; + END IF; + END $clean$; +$remote$); +SELECT pg_temp.assert_engine_gone(engine_id) FROM _e2e73_ids +WHERE origin = '_e2e73_origin' AND scenario IN ('completed', 'failed', 'cancelled'); +DO $$ +BEGIN + IF NOT EXISTS (SELECT 1 FROM df.instances WHERE label = 'e2e73-shadow' AND status = 'running') OR + NOT EXISTS (SELECT 1 FROM df.nodes WHERE id = '00000073') OR + EXISTS (SELECT 1 FROM df.instances WHERE label IN ('e2e73-live', 'e2e73-peer-live', 'e2e73-removed')) THEN + RAISE EXCEPTION 'TEST FAILED [isolation]: satellite retention crossed into control'; + END IF; +END $$; + +DROP TABLE IF EXISTS public._e2e73_success; +CREATE TABLE public._e2e73_success (marker TEXT PRIMARY KEY); +INSERT INTO _e2e73_ids +SELECT current_database(), 'control', local_id, local_id +FROM (SELECT df.start(df.wait_for_signal('release') + ~> 'INSERT INTO public._e2e73_success VALUES (''control'') ON CONFLICT DO NOTHING', + 'e2e73-control') AS local_id) AS root; +SELECT pg_temp.wait_engine(engine_id, 'Running') FROM _e2e73_ids WHERE scenario = 'control'; +SELECT pg_temp.wait_subscription(engine_id, 'release') FROM _e2e73_ids WHERE scenario = 'control'; +SELECT df.signal(local_id, 'release') FROM _e2e73_ids WHERE scenario = 'control'; +DO $$ +DECLARE + control_id TEXT := (SELECT local_id FROM _e2e73_ids WHERE scenario = 'control'); +BEGIN + FOR attempt IN 1..300 LOOP + IF EXISTS (SELECT 1 FROM public._e2e73_success WHERE marker = 'control') THEN + RETURN; + END IF; + PERFORM pg_sleep(0.1); + END LOOP; + RAISE EXCEPTION 'TEST FAILED [control release]: committed success marker missing (engine status: %)', + pg_temp.engine_status(control_id); +END $$; +SELECT pg_temp.wait_engine(engine_id, NULL) FROM _e2e73_ids WHERE scenario = 'control'; + +SELECT dblink_exec('_e2e73_origin', 'BEGIN'); +INSERT INTO _e2e73_ids +SELECT '_e2e73_origin', 'orphan', remote.local_id, remote.prefix || remote.local_id +FROM dblink('_e2e73_origin', $remote$ + SELECT df.start('SELECT 73', 'e2e73-rollback'), + 'pgdf-' || (SELECT oid::text FROM pg_database WHERE datname = current_database()) || + '-' || (SELECT replace(id::text, '-', '') FROM df._installation) || '-' +$remote$) AS remote(local_id TEXT, prefix TEXT); +SELECT pg_temp.wait_engine(engine_id, 'Running') FROM _e2e73_ids WHERE scenario = 'orphan'; +SELECT dblink_exec('_e2e73_origin', 'ROLLBACK'); +SELECT pg_temp.wait_engine(engine_id, NULL) FROM _e2e73_ids WHERE scenario = 'orphan'; +SELECT pg_temp.assert_engine_gone(engine_id) FROM _e2e73_ids WHERE scenario = 'orphan'; + +ALTER DATABASE _e2e73_peer ALLOW_CONNECTIONS false; +SELECT dblink_exec('_e2e73_peer', $remote$ + DROP TRIGGER e2e73_hold_metadata ON df.instances; + UPDATE df.instances SET created_at = clock_timestamp() - interval '1 minute', + completed_at = clock_timestamp() - interval '1 minute' WHERE label = 'e2e73-held-peer'; +$remote$); +SELECT pg_sleep(12); +DO $$ +DECLARE + workflow RECORD; +BEGIN + FOR workflow IN SELECT * FROM _e2e73_ids WHERE origin = '_e2e73_peer' LOOP + IF pg_temp.engine_status(workflow.engine_id) IS DISTINCT FROM + (CASE WHEN workflow.scenario = 'failed' THEN 'Failed' ELSE 'Running' END) THEN + RAISE EXCEPTION 'TEST FAILED [unreachable origin]: engine data changed for %', workflow.scenario; + END IF; + END LOOP; + PERFORM dblink_exec('_e2e73_peer', $remote$ + DO $check$ BEGIN + IF (SELECT count(*) FROM df.instances) <> 2 OR NOT EXISTS (SELECT 1 FROM df.nodes) THEN + RAISE EXCEPTION 'TEST FAILED [unreachable origin]: metadata destroyed'; + END IF; + END $check$; + $remote$); +END $$; +ALTER DATABASE _e2e73_peer ALLOW_CONNECTIONS true; +SELECT pg_temp.wait_engine(engine_id, NULL) FROM _e2e73_ids +WHERE origin = '_e2e73_peer' AND scenario = 'failed'; +SELECT pg_temp.assert_engine_gone(engine_id) FROM _e2e73_ids +WHERE origin = '_e2e73_peer' AND scenario = 'failed'; + +SELECT dblink_exec('_e2e73_peer', 'BEGIN; LOCK TABLE df._installation IN ACCESS EXCLUSIVE MODE'); +DO $$ +DECLARE + saw_blocked_probe BOOLEAN := false; + live_id TEXT := (SELECT engine_id FROM _e2e73_ids WHERE origin = '_e2e73_peer' AND scenario = 'live'); +BEGIN + FOR attempt IN 1..120 LOOP + PERFORM pg_stat_clear_snapshot(); + saw_blocked_probe := saw_blocked_probe OR EXISTS ( + SELECT 1 FROM pg_stat_activity WHERE datname = '_e2e73_peer' + AND application_name = 'pg_durable:worker:management' AND wait_event_type = 'Lock'); + IF (SELECT count(*) FROM pg_stat_activity WHERE datname IN + ('_e2e73_origin', '_e2e73_peer', '_e2e73_removed') + AND application_name LIKE 'pg_durable:worker:%') > + current_setting('pg_durable.max_origin_connections')::int THEN + RAISE EXCEPTION 'TEST FAILED [connection budget]: origin admission limit exceeded'; + END IF; + IF pg_temp.engine_status(live_id) IS DISTINCT FROM 'Running' THEN + RAISE EXCEPTION 'TEST FAILED [locked origin]: live root cancelled or deleted'; + END IF; + PERFORM pg_sleep(0.1); + END LOOP; + IF NOT saw_blocked_probe THEN + RAISE EXCEPTION 'TEST FAILED [locked origin]: maintenance never attempted the locked origin'; + END IF; +END $$; +SELECT dblink_exec('_e2e73_peer', 'ROLLBACK'); +DO $$ +BEGIN + FOR attempt IN 1..300 LOOP + PERFORM pg_stat_clear_snapshot(); + IF NOT EXISTS (SELECT 1 FROM pg_stat_activity WHERE datname IN + ('_e2e73_origin', '_e2e73_peer', '_e2e73_removed') + AND application_name LIKE 'pg_durable:worker:%') THEN + RETURN; + END IF; + PERFORM pg_sleep(0.1); + END LOOP; + RAISE EXCEPTION 'TEST FAILED [connection cleanup]: origin connections did not drain'; +END $$; + +-- Future timestamps exercise the non-expired side of the retention boundary at retention_days=0. +-- Cancellation must ignore creation age; deletion must still honor completion age. +CREATE FUNCTION public.e2e73_hold_engine_terminal() RETURNS trigger LANGUAGE plpgsql AS $$ +BEGIN + IF NEW.instance_id = TG_ARGV[0] AND NEW.status IN ('Completed', 'Failed') THEN + NEW.completed_at := clock_timestamp() + interval '31 days'; + END IF; + RETURN NEW; +END $$; +DO $$ +DECLARE + removed_id TEXT := (SELECT engine_id FROM _e2e73_ids WHERE scenario = 'removed'); +BEGIN + EXECUTE format('UPDATE %I.instances SET created_at = clock_timestamp() + interval ''31 days'' + WHERE instance_id = $1', df.duroxide_schema()) USING removed_id; + EXECUTE format('CREATE TRIGGER e2e73_hold_terminal BEFORE UPDATE OF status ON %I.executions + FOR EACH ROW EXECUTE FUNCTION public.e2e73_hold_engine_terminal(%L)', df.duroxide_schema(), removed_id); +END $$; +SELECT dblink_exec('_e2e73_removed', 'DROP EXTENSION pg_durable CASCADE'); +SELECT pg_temp.wait_engine(engine_id, 'Failed') FROM _e2e73_ids WHERE scenario = 'removed'; +SELECT pg_sleep(8); +DO $$ +DECLARE + removed_id TEXT := (SELECT engine_id FROM _e2e73_ids WHERE scenario = 'removed'); +BEGIN + IF pg_temp.engine_status(removed_id) IS DISTINCT FROM 'Failed' THEN + RAISE EXCEPTION 'TEST FAILED [removed retention]: unexpired terminal data deleted'; + END IF; + EXECUTE format('DROP TRIGGER e2e73_hold_terminal ON %I.executions', df.duroxide_schema()); + EXECUTE format('UPDATE %I.executions SET completed_at = clock_timestamp() - interval ''1 minute'' + WHERE instance_id = $1', df.duroxide_schema()) USING removed_id; +END $$; +DROP FUNCTION public.e2e73_hold_engine_terminal(); +SELECT pg_temp.wait_engine(engine_id, NULL) FROM _e2e73_ids WHERE scenario = 'removed'; +SELECT pg_temp.assert_engine_gone(engine_id) FROM _e2e73_ids WHERE scenario = 'removed'; + +DO $$ +DECLARE + workflow RECORD; +BEGIN + FOR workflow IN SELECT * FROM _e2e73_ids WHERE scenario = 'live' LOOP + IF pg_temp.engine_status(workflow.engine_id) IS DISTINCT FROM 'Running' THEN + RAISE EXCEPTION 'TEST FAILED [live isolation]: running root % was lost', workflow.origin; + END IF; + IF workflow.origin = '_e2e73_peer' THEN + PERFORM pg_temp.wait_subscription(workflow.engine_id, 'release'); + PERFORM dblink_exec(workflow.origin, format( + 'DO $signal$ BEGIN PERFORM df.signal(%L, ''release''); END $signal$;', workflow.local_id)); + END IF; + END LOOP; +END $$; +SELECT dblink_exec('_e2e73_peer', $remote$ + DO $released$ + BEGIN + FOR attempt IN 1..300 LOOP + IF EXISTS (SELECT 1 FROM public._e2e73_success WHERE marker = 'peer') THEN + RETURN; + END IF; + PERFORM pg_sleep(0.1); + END LOOP; + RAISE EXCEPTION 'TEST FAILED [peer release]: committed success marker missing'; + END $released$; +$remote$); + SELECT dblink_disconnect('_e2e73_origin'); + DROP DATABASE _e2e73_origin WITH (FORCE); +SELECT pg_temp.wait_engine(engine_id, NULL) FROM _e2e73_ids WHERE scenario = 'live'; +SELECT pg_temp.assert_engine_gone(engine_id) FROM _e2e73_ids; + +BEGIN; +DELETE FROM df.nodes WHERE instance_id IN (SELECT id FROM df.instances WHERE label = 'e2e73-shadow'); +DELETE FROM df.instances WHERE label = 'e2e73-shadow'; +COMMIT; +DROP TABLE public._e2e73_success; +SELECT dblink_exec('_e2e73_peer', 'DROP TABLE public._e2e73_success'); +SELECT dblink_disconnect(origin_name) +FROM unnest(ARRAY['_e2e73_peer', '_e2e73_removed']) AS origins(origin_name); +DROP DATABASE _e2e73_peer; +DROP DATABASE _e2e73_removed; + +SELECT 'TEST PASSED: multi-database reconciliation' AS result; \ No newline at end of file diff --git a/tests/e2e/sql/74_multi_database_guards.sql b/tests/e2e/sql/74_multi_database_guards.sql new file mode 100644 index 00000000..6c64feec --- /dev/null +++ b/tests/e2e/sql/74_multi_database_guards.sql @@ -0,0 +1,421 @@ +CREATE EXTENSION IF NOT EXISTS dblink; + +DO $$ +BEGIN + IF current_database() IS DISTINCT FROM df.target_database() OR + current_setting('server_version_num')::int < 170000 THEN + RAISE EXCEPTION 'TEST SETUP ERROR: run guards in the PG17+ control database'; + END IF; + IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'df_e2e_user' + AND NOT rolsuper AND NOT rolbypassrls) THEN + RAISE EXCEPTION 'TEST SETUP ERROR: df_e2e_user must not bypass RLS'; + END IF; +END $$; + +DROP DATABASE IF EXISTS _e2e74_origin WITH (FORCE); +CREATE DATABASE _e2e74_origin; +GRANT CONNECT ON DATABASE _e2e74_origin TO df_e2e_user; +CREATE TEMP TABLE _e2e74_epoch AS SELECT epoch_id FROM df._worker_epoch; +CREATE TEMP TABLE _e2e74_connections (connection_name TEXT PRIMARY KEY, pid INT); +CREATE TEMP TABLE _e2e74_state (scenario TEXT PRIMARY KEY, local_id TEXT, engine_id TEXT); +CREATE TEMP TABLE _e2e74_relations (relation_id OID PRIMARY KEY); + +DO $$ +DECLARE + connection_name TEXT; + database_name TEXT; +BEGIN + FOREACH connection_name IN ARRAY ARRAY['e2e74_admin', 'e2e74_submit', 'e2e74_fresh', + 'e2e74_ddl', 'e2e74_gate', 'e2e74_marker', 'e2e74_ready_lock'] LOOP + database_name := CASE WHEN connection_name IN ('e2e74_marker', 'e2e74_ready_lock') + THEN current_database() ELSE '_e2e74_origin' END; + PERFORM dblink_connect(connection_name, format( + 'host=localhost port=%s dbname=%L user=postgres application_name=%s options=%L', + current_setting('port'), database_name, connection_name, + '-c statement_timeout=20000 -c lock_timeout=0 -c idle_in_transaction_session_timeout=0 -c transaction_timeout=0')); + INSERT INTO _e2e74_connections SELECT connection_name, remote.pid + FROM dblink(connection_name, 'SELECT pg_backend_pid()') AS remote(pid INT); + END LOOP; + PERFORM dblink_exec('e2e74_admin', $remote$ + CREATE EXTENSION pg_durable; + DO $grant$ BEGIN PERFORM df.grant_usage('df_e2e_user'); END $grant$; + $remote$); + PERFORM dblink_exec('e2e74_submit', 'SET SESSION AUTHORIZATION df_e2e_user'); + PERFORM dblink_exec('e2e74_fresh', 'SET SESSION AUTHORIZATION df_e2e_user'); +END $$; + +INSERT INTO _e2e74_relations SELECT relation_id FROM dblink('e2e74_admin', + 'SELECT unnest(ARRAY[''df._installation''::regclass::oid, + ''df.instances''::regclass::oid, ''df.nodes''::regclass::oid])') AS remote(relation_id OID); + +CREATE TEMP VIEW _e2e74_guards AS +SELECT activity.pid, activity.xact_start, activity.state_change +FROM pg_stat_activity AS activity +WHERE activity.datname = '_e2e74_origin' + AND activity.application_name = 'pg_durable:worker:management' + AND activity.state = 'idle in transaction' + AND (SELECT count(DISTINCT locks.relation) FROM pg_locks AS locks + WHERE locks.pid = activity.pid AND locks.database = activity.datid + AND locks.relation IN (SELECT relation_id FROM _e2e74_relations) + AND locks.mode = 'AccessShareLock' AND locks.granted) = 3; + +CREATE FUNCTION pg_temp.e2e74_wait(check_sql TEXT, assertion TEXT, seconds INT DEFAULT 10) +RETURNS void LANGUAGE plpgsql AS $$ +DECLARE + deadline TIMESTAMPTZ := clock_timestamp() + make_interval(secs => seconds); + satisfied BOOLEAN; +BEGIN + LOOP + PERFORM pg_stat_clear_snapshot(); + EXECUTE check_sql INTO satisfied; + IF satisfied IS TRUE THEN + RETURN; + END IF; + IF clock_timestamp() >= deadline THEN + RAISE EXCEPTION 'TEST FAILED [%]: condition not observed within %s', assertion, seconds; + END IF; + PERFORM pg_sleep(0.025); + END LOOP; +END $$; + +DROP FUNCTION IF EXISTS public.e2e74_effect(INT); +DROP TABLE IF EXISTS public.e2e74_effects; +CREATE TABLE public.e2e74_effects ( + marker INT PRIMARY KEY, + role_name TEXT NOT NULL DEFAULT current_user, + database_name TEXT NOT NULL DEFAULT current_database() +); +GRANT INSERT, SELECT ON public.e2e74_effects TO df_e2e_user; +CREATE FUNCTION public.e2e74_effect(marker_key INT) RETURNS INT LANGUAGE plpgsql AS $$ +BEGIN + PERFORM pg_advisory_xact_lock(740074, marker_key); + INSERT INTO public.e2e74_effects (marker) VALUES (marker_key); + RETURN marker_key; +END $$; + +CREATE FUNCTION pg_temp.e2e74_start(scenario_name TEXT, marker_key INT, + submit_connection TEXT DEFAULT 'e2e74_submit') +RETURNS void LANGUAGE plpgsql AS $$ +BEGIN + INSERT INTO _e2e74_state + SELECT scenario_name, remote.local_id, remote.prefix || remote.local_id + FROM dblink(submit_connection, format($remote$ + SELECT df.start(%L, %L, database => %L), + 'pgdf-' || (SELECT oid::text FROM pg_database WHERE datname = current_database()) || + '-' || (SELECT replace(id::text, '-', '') FROM df._installation) || '-' + $remote$, format('SELECT public.e2e74_effect(%s)', marker_key), + 'e2e74-' || scenario_name, current_database())) AS remote(local_id TEXT, prefix TEXT); +END $$; + +CREATE FUNCTION pg_temp.e2e74_completed(scenario_name TEXT) RETURNS void LANGUAGE plpgsql AS $$ +DECLARE + instance_id TEXT := (SELECT local_id FROM _e2e74_state WHERE scenario = scenario_name); + workflow_engine_id TEXT := (SELECT engine_id FROM _e2e74_state WHERE scenario = scenario_name); +BEGIN + PERFORM pg_temp.e2e74_wait(format( + 'SELECT EXISTS (SELECT 1 FROM %I.get_instance_info(%L) WHERE status = ''Completed'')', + df.duroxide_schema(), workflow_engine_id), scenario_name || ': engine completed', 15); + PERFORM pg_temp.e2e74_wait(format($check$ + SELECT remote.completed FROM dblink('e2e74_admin', %L) AS remote(completed BOOLEAN) + $check$, format($remote$ + SELECT EXISTS (SELECT 1 FROM df.instance_info(%1$L) WHERE status = 'completed') + AND EXISTS (SELECT 1 FROM df.instances WHERE id = %1$L AND status = 'completed') + AND EXISTS (SELECT 1 FROM df.nodes WHERE instance_id = %1$L AND status = 'completed') + $remote$, instance_id)), scenario_name || ': retained engine and metadata completed', 15); +END $$; + +CREATE FUNCTION pg_temp.e2e74_drain() RETURNS void LANGUAGE plpgsql AS $$ +BEGIN + PERFORM pg_temp.e2e74_wait($check$ + SELECT NOT EXISTS (SELECT 1 FROM pg_stat_activity WHERE datname = '_e2e74_origin' + AND application_name LIKE 'pg_durable:worker:%') + $check$, 'origin connections and guard transactions drained', 15); +END $$; + +CREATE FUNCTION pg_temp.e2e74_guard_case(scenario_name TEXT, marker_key INT, hold_seconds INT) +RETURNS void LANGUAGE plpgsql AS $$ +DECLARE + guard_pid INT; + ddl_pid INT := (SELECT pid FROM _e2e74_connections WHERE connection_name = 'e2e74_ddl'); + deadline TIMESTAMPTZ; + ddl_result TEXT; +BEGIN + PERFORM pg_temp.e2e74_drain(); + PERFORM dblink_exec('e2e74_marker', format( + 'BEGIN; DO $hold$ BEGIN PERFORM pg_advisory_xact_lock(740074, %s); END $hold$', marker_key)); + PERFORM pg_temp.e2e74_start(scenario_name, marker_key); + PERFORM pg_temp.e2e74_wait(format($check$ + SELECT EXISTS (SELECT 1 FROM pg_locks AS locks JOIN pg_stat_activity AS activity USING (pid) + WHERE locks.locktype = 'advisory' AND locks.classid = 740074 AND locks.objid = %s + AND locks.objsubid = 2 AND NOT locks.granted + AND activity.datname = current_database() AND activity.usename = 'df_e2e_user') + AND EXISTS (SELECT 1 FROM _e2e74_guards) + $check$, marker_key), scenario_name || ': SQL waiting with separate idle metadata guard'); + SELECT pid INTO STRICT guard_pid FROM _e2e74_guards; + IF dblink_send_query('e2e74_ddl', format( + 'ALTER TABLE df.instances ADD COLUMN e2e74_guard_%s INT', marker_key)) <> 1 THEN + RAISE EXCEPTION 'TEST FAILED [%]: could not queue DDL', scenario_name; + END IF; + PERFORM pg_temp.e2e74_wait(format($check$ + SELECT EXISTS (SELECT 1 FROM pg_locks WHERE pid = %1$s + AND locktype = 'relation' AND mode = 'AccessExclusiveLock' AND NOT granted) + AND %2$s = ANY(pg_blocking_pids(%1$s)) + $check$, ddl_pid, guard_pid), scenario_name || ': DDL queued AFTER guard acquisition'); + + deadline := clock_timestamp() + make_interval(secs => hold_seconds); + LOOP + PERFORM pg_stat_clear_snapshot(); + IF dblink_is_busy('e2e74_ddl') <> 1 OR + NOT EXISTS (SELECT 1 FROM _e2e74_guards WHERE pid = guard_pid) OR + NOT (guard_pid = ANY(pg_blocking_pids(ddl_pid))) OR + EXISTS (SELECT 1 FROM public.e2e74_effects WHERE marker = marker_key) THEN + RAISE EXCEPTION 'TEST FAILED [%]: guard lost or DDL/effect ran before SQL release', scenario_name; + END IF; + EXIT WHEN clock_timestamp() >= deadline; + PERFORM pg_sleep(0.025); + END LOOP; + + PERFORM dblink_exec('e2e74_marker', 'COMMIT'); + PERFORM pg_temp.e2e74_wait('SELECT dblink_is_busy(''e2e74_ddl'') = 0', + scenario_name || ': queued DDL finishes after SQL release', 8); + SELECT status INTO ddl_result FROM dblink_get_result('e2e74_ddl') AS remote(status TEXT); + IF ddl_result IS DISTINCT FROM 'ALTER TABLE' THEN + RAISE EXCEPTION 'TEST FAILED [%]: DDL result = %', scenario_name, ddl_result; + END IF; + PERFORM status FROM dblink_get_result('e2e74_ddl') AS remote(status TEXT); + PERFORM pg_temp.e2e74_completed(scenario_name); + IF NOT EXISTS (SELECT 1 FROM public.e2e74_effects WHERE marker = marker_key + AND role_name = 'df_e2e_user' AND database_name = current_database()) THEN + RAISE EXCEPTION 'TEST FAILED [%]: committed effect missing or wrong execution identity', scenario_name; + END IF; + PERFORM pg_temp.e2e74_drain(); +END $$; + +-- Guard first, queued ACCESS EXCLUSIVE second, SQL completion last: not DDL-first admission. +SELECT pg_temp.e2e74_guard_case('reverse-order', 1, 0); +SELECT pg_temp.e2e74_start('after-ddl', 11); +SELECT pg_temp.e2e74_completed('after-ddl'); +SELECT pg_temp.e2e74_drain(); + +-- Pause the final status route's second connection AFTER its guard, BEFORE its UPDATE. +-- Login's own transaction releases its locks before the metadata query needs them again. +SELECT dblink_exec('e2e74_gate', + 'BEGIN; DO $hold$ BEGIN PERFORM pg_advisory_xact_lock(740074, 74); END $hold$'); +SELECT dblink_exec('e2e74_admin', $remote$ + CREATE FUNCTION public.e2e74_second_connection() RETURNS event_trigger LANGUAGE plpgsql AS $gate$ + BEGIN + IF current_setting('application_name') = 'pg_durable:worker:management' AND + EXISTS (SELECT 1 FROM pg_stat_activity AS activity JOIN pg_locks AS locks USING (pid) + WHERE activity.datid = (SELECT oid FROM pg_database WHERE datname = current_database()) + AND activity.application_name = 'pg_durable:worker:management' + AND activity.state = 'idle in transaction' AND activity.pid <> pg_backend_pid() + AND locks.relation = 'df.instances'::regclass + AND locks.mode = 'AccessShareLock' AND locks.granted) AND + EXISTS (SELECT 1 FROM df.instances AS instance WHERE instance.label = 'e2e74-reverse-query' + AND instance.status = 'running' + AND EXISTS (SELECT 1 FROM df.nodes WHERE instance_id = instance.id) + AND NOT EXISTS (SELECT 1 FROM df.nodes WHERE instance_id = instance.id + AND status IS DISTINCT FROM 'completed')) THEN + PERFORM pg_advisory_xact_lock(740074, 74); + END IF; + END $gate$; + CREATE EVENT TRIGGER e2e74_second_connection ON login + EXECUTE FUNCTION public.e2e74_second_connection(); +$remote$); +SELECT pg_temp.e2e74_start('reverse-query', 4); +SELECT pg_temp.e2e74_wait($check$ + SELECT EXISTS (SELECT 1 FROM pg_locks AS locks JOIN pg_stat_activity AS activity USING (pid) + WHERE activity.datname = '_e2e74_origin' + AND activity.application_name = 'pg_durable:worker:management' + AND locks.locktype = 'advisory' AND locks.classid = 740074 AND locks.objid = 74 + AND locks.objsubid = 2 AND NOT locks.granted) + AND EXISTS (SELECT 1 FROM _e2e74_guards) +$check$, 'final status route paused before second connection query'); +DO $$ +DECLARE + guard_pid INT := (SELECT pid FROM _e2e74_guards); + second_pid INT := (SELECT locks.pid FROM pg_locks AS locks JOIN pg_stat_activity AS activity USING (pid) + WHERE activity.datname = '_e2e74_origin' AND locks.locktype = 'advisory' + AND locks.classid = 740074 AND locks.objid = 74 AND locks.objsubid = 2 AND NOT locks.granted); + ddl_pid INT := (SELECT pid FROM _e2e74_connections WHERE connection_name = 'e2e74_ddl'); + ddl_result TEXT; +BEGIN + IF dblink_send_query('e2e74_ddl', 'ALTER TABLE df.instances ADD COLUMN e2e74_second_query INT') <> 1 THEN + RAISE EXCEPTION 'TEST FAILED [reverse query]: could not queue DDL'; + END IF; + PERFORM pg_temp.e2e74_wait(format($check$ + SELECT %1$s = ANY(pg_blocking_pids(%2$s)) AND EXISTS (SELECT 1 FROM pg_locks + WHERE pid = %2$s AND mode = 'AccessExclusiveLock' AND NOT granted) + $check$, guard_pid, ddl_pid), 'second-query DDL queued behind existing guard'); + PERFORM dblink_exec('e2e74_gate', 'COMMIT'); + PERFORM pg_temp.e2e74_wait(format($check$ + SELECT EXISTS (SELECT 1 FROM pg_stat_activity WHERE pid = %1$s + AND query LIKE 'UPDATE df.instances%%' AND wait_event_type = 'Lock') + AND %2$s = ANY(pg_blocking_pids(%1$s)) + AND EXISTS (SELECT 1 FROM _e2e74_guards WHERE pid = %3$s) + AND %3$s = ANY(pg_blocking_pids(%2$s)) + $check$, second_pid, ddl_pid, guard_pid), 'observed second UPDATE -> queued DDL -> idle guard', 3); + PERFORM pg_temp.e2e74_wait('SELECT dblink_is_busy(''e2e74_ddl'') = 0', + 'second query deadline breaks client-side lock cycle', 8); + SELECT status INTO ddl_result FROM dblink_get_result('e2e74_ddl') AS remote(status TEXT); + IF ddl_result IS DISTINCT FROM 'ALTER TABLE' THEN + RAISE EXCEPTION 'TEST FAILED [reverse query]: DDL result = %', ddl_result; + END IF; + PERFORM status FROM dblink_get_result('e2e74_ddl') AS remote(status TEXT); +END $$; +SELECT dblink_exec('e2e74_admin', + 'DROP EVENT TRIGGER e2e74_second_connection; DROP FUNCTION public.e2e74_second_connection()'); +SELECT pg_temp.e2e74_completed('reverse-query'); +SELECT pg_temp.e2e74_drain(); + +-- Block the completion UPDATE on the second pool connection, then cancel as its row-lock owner. +SELECT dblink_exec('e2e74_marker', + 'BEGIN; DO $hold$ BEGIN PERFORM pg_advisory_xact_lock(740074, 2); END $hold$'); +SELECT pg_temp.e2e74_start('cancel-drain', 2); +SELECT pg_temp.e2e74_wait($check$ + SELECT EXISTS (SELECT 1 FROM pg_locks AS locks JOIN pg_stat_activity AS activity USING (pid) + WHERE locks.locktype = 'advisory' AND locks.classid = 740074 AND locks.objid = 2 + AND locks.objsubid = 2 AND NOT locks.granted + AND activity.usename = 'df_e2e_user' AND activity.datname = current_database()) +$check$, 'cancellation SQL reached marker'); +DO $$ +DECLARE + instance_id TEXT := (SELECT local_id FROM _e2e74_state WHERE scenario = 'cancel-drain'); +BEGIN + PERFORM dblink_exec('e2e74_submit', format($remote$ + BEGIN; + DO $lock$ BEGIN PERFORM id FROM df.instances WHERE id = %L FOR UPDATE; END $lock$; + $remote$, instance_id)); +END $$; +SELECT dblink_exec('e2e74_marker', 'COMMIT'); +SELECT pg_temp.e2e74_wait($check$ + SELECT EXISTS (SELECT 1 FROM pg_stat_activity AS activity + WHERE activity.datname = '_e2e74_origin' + AND activity.application_name = 'pg_durable:worker:management' + AND activity.query LIKE 'UPDATE df.instances%' + AND activity.wait_event_type = 'Lock' + AND (SELECT pid FROM _e2e74_connections WHERE connection_name = 'e2e74_submit') + = ANY(pg_blocking_pids(activity.pid)) + AND EXISTS (SELECT 1 FROM _e2e74_guards WHERE pid <> activity.pid)) +$check$, 'status UPDATE blocked on submitting role, separate guard already acquired'); + +DO $$ +DECLARE + instance_id TEXT := (SELECT local_id FROM _e2e74_state WHERE scenario = 'cancel-drain'); + engine_id TEXT := (SELECT state.engine_id FROM _e2e74_state AS state WHERE scenario = 'cancel-drain'); + cancel_result TEXT; +BEGIN + SELECT result INTO cancel_result FROM dblink('e2e74_submit', format( + 'SELECT df.cancel(%L, ''e2e74 drain'')', instance_id)) AS remote(result TEXT); + IF cancel_result IS DISTINCT FROM format('Instance %s cancelled: e2e74 drain', instance_id) THEN + RAISE EXCEPTION 'TEST FAILED [cancel]: %', cancel_result; + END IF; + PERFORM pg_temp.e2e74_wait(format( + 'SELECT EXISTS (SELECT 1 FROM %I.get_instance_info(%L) WHERE status = ''Failed'')', + df.duroxide_schema(), engine_id), 'cancelled engine retained in Failed state', 15); +END $$; +SELECT pg_temp.e2e74_drain(); +SELECT pg_temp.e2e74_start('permits-reusable', 12, 'e2e74_fresh'); +SELECT pg_temp.e2e74_completed('permits-reusable'); +SELECT pg_temp.e2e74_drain(); +DO $$ +DECLARE + submit_pid INT := (SELECT pid FROM _e2e74_connections WHERE connection_name = 'e2e74_submit'); +BEGIN + PERFORM pg_stat_clear_snapshot(); + IF NOT EXISTS (SELECT 1 FROM pg_stat_activity WHERE pid = submit_pid AND state = 'idle in transaction') OR + NOT EXISTS (SELECT 1 FROM pg_locks WHERE pid = submit_pid + AND locktype = 'transactionid' AND mode = 'ExclusiveLock' AND granted) OR + NOT EXISTS (SELECT 1 FROM public.e2e74_effects WHERE marker = 2) THEN + RAISE EXCEPTION 'TEST FAILED [drain]: row-lock transaction ended early or SQL effect missing'; + END IF; + PERFORM dblink_exec('e2e74_admin', format($remote$ + DO $locked$ BEGIN + BEGIN + PERFORM id FROM df.instances WHERE id = %L FOR UPDATE NOWAIT; + RAISE EXCEPTION 'TEST FAILED [drain]: cancelled instance row lock was released early'; + EXCEPTION WHEN lock_not_available THEN NULL; + END; + END $locked$; + $remote$, (SELECT local_id FROM _e2e74_state WHERE scenario = 'cancel-drain'))); +END $$; +SELECT dblink_exec('e2e74_submit', 'COMMIT'); +SELECT pg_temp.e2e74_wait(format( + 'SELECT remote.cancelled FROM dblink(''e2e74_admin'', %L) AS remote(cancelled BOOLEAN)', + format('SELECT df.status(%L) = ''cancelled''', local_id)), 'cancelled metadata remains terminal') +FROM _e2e74_state WHERE scenario = 'cancel-drain'; + +-- All worker origin connections are fresh; admin sessions override these DB defaults. +-- SQL explicitly targets control, so a user-session timeout cannot impersonate a dead guard. +ALTER DATABASE _e2e74_origin SET idle_in_transaction_session_timeout = '1s'; +ALTER DATABASE _e2e74_origin SET transaction_timeout = '2s'; +SELECT pg_temp.e2e74_guard_case('guard-timeouts', 3, 3); +SELECT pg_temp.e2e74_start('after-timeouts', 13); +SELECT pg_temp.e2e74_completed('after-timeouts'); +SELECT pg_temp.e2e74_drain(); + +-- Readiness resolution must time out on the control query, before the harness's 20s timeout. +DO $$ +DECLARE + instance_id TEXT := (SELECT local_id FROM _e2e74_state WHERE scenario = 'after-timeouts'); +BEGIN + PERFORM dblink_exec('e2e74_ready_lock', format( + 'BEGIN; LOCK TABLE %I._worker_ready IN ACCESS EXCLUSIVE MODE', df.duroxide_schema())); + IF dblink_send_query('e2e74_submit', format( + 'SELECT count(*)::text FROM df.instance_info(%L)', instance_id)) <> 1 THEN + RAISE EXCEPTION 'TEST FAILED [readiness]: could not send management query'; + END IF; +END $$; +SELECT pg_temp.e2e74_wait($check$ + SELECT EXISTS (SELECT 1 FROM pg_stat_activity AS activity + WHERE activity.datname = current_database() AND activity.query LIKE '%_worker_ready%' + AND activity.wait_event_type = 'Lock' + AND (SELECT pid FROM _e2e74_connections WHERE connection_name = 'e2e74_ready_lock') + = ANY(pg_blocking_pids(activity.pid))) +$check$, 'satellite management blocked on control readiness', 2); +SELECT pg_temp.e2e74_wait('SELECT dblink_is_busy(''e2e74_submit'') = 0', + 'control readiness lookup has a server-side deadline', 4); +DO $$ +DECLARE + error_message TEXT; + returned_rows INT; +BEGIN + SELECT count(*) INTO returned_rows FROM dblink_get_result('e2e74_submit', false) AS remote(result TEXT); + error_message := dblink_error_message('e2e74_submit'); + IF returned_rows <> 0 OR error_message NOT LIKE '%control installation unavailable%' OR + error_message !~ '(lock timeout|statement timeout)' THEN + RAISE EXCEPTION 'TEST FAILED [readiness]: expected bounded control error, rows=%, error=%', + returned_rows, error_message; + END IF; + PERFORM result FROM dblink_get_result('e2e74_submit', false) AS remote(result TEXT); + IF NOT EXISTS (SELECT 1 FROM pg_locks WHERE pid = + (SELECT pid FROM _e2e74_connections WHERE connection_name = 'e2e74_ready_lock') + AND mode = 'AccessExclusiveLock' AND granted) THEN + RAISE EXCEPTION 'TEST FAILED [readiness]: blocker ended before deadline assertion'; + END IF; +END $$; +SELECT dblink_exec('e2e74_ready_lock', 'ROLLBACK'); +SELECT pg_temp.e2e74_completed('after-timeouts'); +SELECT pg_temp.e2e74_start('after-readiness', 14); +SELECT pg_temp.e2e74_completed('after-readiness'); +SELECT pg_temp.e2e74_drain(); + +DO $$ +BEGIN + IF (SELECT count(*) FROM public.e2e74_effects) <> 8 OR + EXISTS (SELECT 1 FROM public.e2e74_effects + WHERE role_name <> 'df_e2e_user' OR database_name <> current_database()) OR + (SELECT count(*) FROM _e2e74_epoch) <> 1 OR + (SELECT epoch_id FROM df._worker_epoch) IS DISTINCT FROM (SELECT epoch_id FROM _e2e74_epoch) THEN + RAISE EXCEPTION 'TEST FAILED [isolation]: missing effects, wrong identity, or control runtime restarted'; + END IF; +END $$; + +SELECT dblink_disconnect(connection_name) FROM _e2e74_connections; +DROP DATABASE _e2e74_origin WITH (FORCE); +DROP FUNCTION public.e2e74_effect(INT); +DROP TABLE public.e2e74_effects; +DROP VIEW _e2e74_guards; +DROP TABLE _e2e74_connections, _e2e74_state, _e2e74_relations, _e2e74_epoch; + +SELECT 'TEST PASSED: multi-database guards' AS result; \ No newline at end of file diff --git a/tests/e2e/sql/75_multi_database_force_drop.sql b/tests/e2e/sql/75_multi_database_force_drop.sql new file mode 100644 index 00000000..51201c7a --- /dev/null +++ b/tests/e2e/sql/75_multi_database_force_drop.sql @@ -0,0 +1,341 @@ +CREATE EXTENSION IF NOT EXISTS dblink; + +DO $$ +BEGIN + IF current_database() IS DISTINCT FROM df.target_database() OR + current_setting('server_version_num')::int < 170000 OR + current_setting('pg_durable.max_user_connections')::int <> 1 OR + current_setting('pg_durable.execution_acquire_timeout')::int <> 30 THEN + RAISE EXCEPTION 'TEST SETUP ERROR: run in the PG17+ control database, force-drop phase (1 slot, 30s admission)'; + END IF; + IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'df_e2e_user' + AND rolcanlogin AND NOT rolsuper AND NOT rolbypassrls) THEN + RAISE EXCEPTION 'TEST SETUP ERROR: df_e2e_user must be a non-superuser login without BYPASSRLS'; + END IF; +END $$; + +DROP DATABASE IF EXISTS _e2e75_origin WITH (FORCE); +CREATE DATABASE _e2e75_origin; +GRANT CONNECT ON DATABASE _e2e75_origin TO df_e2e_user; + +DROP TABLE IF EXISTS public.e2e75_effects; +CREATE TABLE public.e2e75_effects ( + marker TEXT PRIMARY KEY, + database_name TEXT NOT NULL DEFAULT current_database(), + role_name TEXT NOT NULL DEFAULT current_user +); +GRANT SELECT, INSERT ON public.e2e75_effects TO df_e2e_user; +CREATE TEMP TABLE _e2e75_control (local_id TEXT PRIMARY KEY, marker TEXT NOT NULL); +GRANT SELECT, INSERT ON _e2e75_control TO df_e2e_user; +CREATE TEMP TABLE _e2e75_unfenced (local_id TEXT PRIMARY KEY); +GRANT SELECT, INSERT ON _e2e75_unfenced TO df_e2e_user; +CREATE TEMP TABLE _e2e75_old ( + local_id TEXT NOT NULL, + engine_id TEXT NOT NULL, + database_oid OID NOT NULL, + installation_id UUID NOT NULL, + submitted_at TIMESTAMPTZ NOT NULL, + guard_pid INT, + released_at TIMESTAMPTZ +); +CREATE TEMP TABLE _e2e75_relations (relation_id OID PRIMARY KEY); +CREATE TEMP TABLE _e2e75_epoch AS SELECT epoch_id FROM df._worker_epoch; + +CREATE FUNCTION pg_temp.e2e75_wait(check_sql TEXT, assertion TEXT, + deadline TIMESTAMPTZ DEFAULT clock_timestamp() + interval '10 seconds') +RETURNS void LANGUAGE plpgsql AS $$ +DECLARE + satisfied BOOLEAN; +BEGIN + LOOP + PERFORM pg_stat_clear_snapshot(); + EXECUTE check_sql INTO satisfied; + IF clock_timestamp() >= deadline THEN + RAISE NOTICE 'Activity at deadline: %', (SELECT jsonb_agg(to_jsonb(activity)) FROM ( + SELECT pid, datname, usename, application_name, state, wait_event_type, + wait_event, pg_blocking_pids(pid) AS blockers, query + FROM pg_stat_activity WHERE pid <> pg_backend_pid() + ) AS activity); + RAISE NOTICE 'Locks at deadline: %', (SELECT jsonb_agg(to_jsonb(locks)) FROM ( + SELECT pid, locktype, database, relation, mode, granted, classid, objid + FROM pg_locks WHERE pid IN (SELECT pid FROM pg_stat_activity + WHERE application_name LIKE 'pg_durable:%' OR application_name LIKE 'e2e75%') + ) AS locks); + RAISE EXCEPTION 'TEST FAILED [%]: deadline exceeded', assertion; + END IF; + IF satisfied IS TRUE THEN + RETURN; + END IF; + PERFORM pg_sleep(0.025); + END LOOP; +END $$; + +DO $$ +BEGIN + PERFORM dblink_connect('e2e75_blocker', format( + 'host=localhost port=%s dbname=%L user=postgres application_name=e2e75_blocker options=%L', + current_setting('port'), current_database(), + '-c idle_in_transaction_session_timeout=0 -c transaction_timeout=0')); + PERFORM dblink_connect('e2e75_origin', format( + 'host=localhost port=%s dbname=_e2e75_origin user=postgres application_name=e2e75_origin', + current_setting('port'))); +END $$; +SELECT dblink_exec('e2e75_origin', $remote$ + CREATE EXTENSION pg_durable; + DO $grant$ BEGIN PERFORM df.grant_usage('df_e2e_user'); END $grant$; + CREATE TABLE public.e2e75_effects ( + marker TEXT PRIMARY KEY, + database_name TEXT NOT NULL DEFAULT current_database(), + role_name TEXT NOT NULL DEFAULT current_user + ); + GRANT SELECT, INSERT ON public.e2e75_effects TO df_e2e_user; + SET SESSION AUTHORIZATION df_e2e_user; +$remote$); +INSERT INTO _e2e75_relations SELECT relation_id FROM dblink('e2e75_origin', + 'SELECT unnest(ARRAY[''df._installation''::regclass::oid, + ''df.instances''::regclass::oid, ''df.nodes''::regclass::oid])') AS remote(relation_id OID); + +CREATE TEMP VIEW _e2e75_waiters AS +SELECT activity.pid +FROM pg_stat_activity AS activity JOIN pg_locks AS locks USING (pid) +WHERE activity.datname = current_database() + AND activity.usename = 'df_e2e_user' + AND activity.application_name = 'pg_durable:worker:workflow-sql' + AND activity.state = 'active' AND activity.wait_event_type = 'Lock' + AND locks.locktype = 'advisory' AND locks.classid = 750075 AND locks.objid = 1 + AND locks.objsubid = 2 AND NOT locks.granted; + +CREATE TEMP VIEW _e2e75_guards AS +SELECT activity.pid +FROM pg_stat_activity AS activity +WHERE activity.datid = (SELECT database_oid FROM _e2e75_old) + AND activity.application_name = 'pg_durable:worker:management' + AND activity.state = 'idle in transaction' + AND (SELECT count(DISTINCT locks.relation) FROM pg_locks AS locks + WHERE locks.pid = activity.pid AND locks.database = activity.datid + AND locks.relation IN (SELECT relation_id FROM _e2e75_relations) + AND locks.mode = 'AccessShareLock' AND locks.granted) = 3; + +DO $$ +BEGIN + EXECUTE format($view$ + CREATE TEMP VIEW _e2e75_sql_history AS + SELECT scheduled.instance_id, scheduled.execution_id, scheduled.event_id, + outcome.event_data::jsonb->>'type' AS outcome_type, + outcome.event_data::jsonb #>> '{details,Application,message}' AS error_message + FROM %1$I.history AS scheduled JOIN _e2e75_old AS old + ON split_part(scheduled.instance_id, '::', 1) = old.engine_id + LEFT JOIN %1$I.history AS outcome + ON outcome.instance_id = scheduled.instance_id + AND outcome.execution_id = scheduled.execution_id + AND outcome.event_data::jsonb->>'source_event_id' = scheduled.event_id::text + AND outcome.event_data::jsonb->>'type' IN ('ActivityCompleted', 'ActivityFailed') + WHERE scheduled.event_data::jsonb->>'type' = 'ActivityScheduled' + AND scheduled.event_data::jsonb->>'name' = 'pg_durable::activity::execute-sql' + $view$, df.duroxide_schema()); +END $$; + +-- One blocker fills the user semaphore but leaves the second activity worker free to route satellite SQL. +SELECT dblink_exec('e2e75_blocker', + 'BEGIN; DO $hold$ BEGIN PERFORM pg_advisory_xact_lock(750075, 1); END $hold$'); +SET SESSION AUTHORIZATION df_e2e_user; +INSERT INTO _e2e75_control SELECT df.start( + 'WITH gate AS MATERIALIZED (SELECT pg_advisory_xact_lock(750075, 1)) + INSERT INTO public.e2e75_effects (marker) SELECT ''control-1'' FROM gate', + 'e2e75-control-1'), 'control-1'; +RESET SESSION AUTHORIZATION; +SELECT pg_temp.e2e75_wait('SELECT count(*) = 1 FROM _e2e75_waiters', 'control SQL permit occupied'); + +DO $$ +DECLARE + started_at TIMESTAMPTZ := clock_timestamp(); +BEGIN + INSERT INTO _e2e75_old (local_id, engine_id, database_oid, installation_id, submitted_at) + SELECT remote.local_id, format('pgdf-%s-%s-%s', remote.database_oid, + replace(remote.installation_id::text, '-', ''), remote.local_id), + remote.database_oid, remote.installation_id, started_at + FROM dblink('e2e75_origin', $remote$ + SELECT df.start('INSERT INTO public.e2e75_effects (marker) VALUES (''old'')', 'e2e75-old'), + (SELECT oid FROM pg_database WHERE datname = current_database()), + (SELECT id FROM df._installation WHERE singleton) + $remote$) AS remote(local_id TEXT, database_oid OID, installation_id UUID); +END $$; +SELECT dblink_disconnect('e2e75_origin'); + +SELECT pg_temp.e2e75_wait($check$ + SELECT (SELECT count(*) FROM _e2e75_waiters) = 1 + AND (SELECT count(*) FROM _e2e75_guards) = 1 + AND NOT EXISTS (SELECT 1 FROM pg_stat_activity WHERE datname = '_e2e75_origin' + AND application_name = 'pg_durable:worker:workflow-sql') + AND (SELECT count(*) = 1 AND bool_and(outcome_type IS NULL) FROM _e2e75_sql_history) +$check$, 'old SQL scheduled with idle origin guard, before its user connection', + (SELECT submitted_at + interval '10 seconds' FROM _e2e75_old)); +UPDATE _e2e75_old SET guard_pid = (SELECT pid FROM _e2e75_guards); + +-- Explicit cross-database control work retains the legacy name-only execution path. +SET SESSION AUTHORIZATION df_e2e_user; +INSERT INTO _e2e75_unfenced SELECT df.start( + 'INSERT INTO public.e2e75_effects (marker) VALUES (''unfenced-old'')', + 'e2e75-unfenced-old', database => '_e2e75_origin'); +RESET SESSION AUTHORIZATION; + +-- FORCE kills the already-acquired guard; the next user connection resolves the reused name. +DROP DATABASE _e2e75_origin WITH (FORCE); +DO $$ +BEGIN + PERFORM pg_stat_clear_snapshot(); + IF EXISTS (SELECT 1 FROM pg_stat_activity WHERE pid = (SELECT guard_pid FROM _e2e75_old)) OR + EXISTS (SELECT 1 FROM pg_database WHERE oid = (SELECT database_oid FROM _e2e75_old)) THEN + RAISE EXCEPTION 'TEST FAILED [force drop]: old database or guard survived'; + END IF; +END $$; +CREATE DATABASE _e2e75_origin; +GRANT CONNECT ON DATABASE _e2e75_origin TO df_e2e_user; +DO $$ +BEGIN + PERFORM dblink_connect('e2e75_origin', format( + 'host=localhost port=%s dbname=_e2e75_origin user=postgres application_name=e2e75_origin', + current_setting('port'))); +END $$; +SELECT dblink_exec('e2e75_origin', $remote$ + CREATE EXTENSION pg_durable; + DO $grant$ BEGIN PERFORM df.grant_usage('df_e2e_user'); END $grant$; + CREATE TABLE public.e2e75_effects ( + marker TEXT PRIMARY KEY, + database_name TEXT NOT NULL DEFAULT current_database(), + role_name TEXT NOT NULL DEFAULT current_user + ); + GRANT SELECT, INSERT ON public.e2e75_effects TO df_e2e_user; + SET SESSION AUTHORIZATION df_e2e_user; +$remote$); +CREATE TEMP TABLE _e2e75_replacement AS +SELECT * FROM dblink('e2e75_origin', $remote$ + SELECT (SELECT oid FROM pg_database WHERE datname = current_database()), + (SELECT id FROM df._installation WHERE singleton), + (SELECT extversion FROM pg_extension WHERE extname = 'pg_durable') +$remote$) AS remote(database_oid OID, installation_id UUID, extension_version TEXT); + +DO $$ +BEGIN + PERFORM pg_stat_clear_snapshot(); + IF (SELECT count(*) FROM _e2e75_waiters) <> 1 OR + EXISTS (SELECT 1 FROM _e2e75_sql_history WHERE outcome_type IS NOT NULL) OR + EXISTS (SELECT 1 FROM pg_stat_activity WHERE datname = '_e2e75_origin' + AND application_name = 'pg_durable:worker:workflow-sql') THEN + RAISE EXCEPTION 'TEST FAILED [replacement setup]: user SQL was admitted before blocker release'; + END IF; + IF NOT EXISTS (SELECT 1 FROM _e2e75_replacement AS replacement CROSS JOIN _e2e75_old AS old + WHERE replacement.database_oid <> old.database_oid + AND replacement.installation_id <> old.installation_id + AND replacement.extension_version = (SELECT extversion FROM pg_extension WHERE extname = 'pg_durable')) THEN + RAISE EXCEPTION 'TEST FAILED [replacement identity]: expected new OID/UUID with current extension version'; + END IF; +END $$; +SELECT dblink_exec('e2e75_blocker', 'COMMIT'); +UPDATE _e2e75_old SET released_at = clock_timestamp(); +DO $$ +BEGIN + IF (SELECT released_at - submitted_at FROM _e2e75_old) >= interval '25 seconds' THEN + RAISE EXCEPTION 'TEST FAILED [admission window]: blocker released too late for the 30s SQL deadline'; + END IF; +END $$; +SELECT dblink_disconnect('e2e75_blocker'); + +-- Inspect execute-sql itself: final status updates retry against the removed origin for ~90s. +SELECT pg_temp.e2e75_wait($check$ + SELECT EXISTS (SELECT 1 FROM _e2e75_sql_history WHERE outcome_type IS NOT NULL) +$check$, 'old execute-sql outcome persisted', + (SELECT released_at + interval '15 seconds' FROM _e2e75_old)); +DO $$ +BEGIN + IF NOT EXISTS (SELECT 1 FROM _e2e75_sql_history WHERE outcome_type = 'ActivityFailed' + AND error_message = 'Origin installation removed or replaced') OR + EXISTS (SELECT 1 FROM _e2e75_sql_history WHERE outcome_type = 'ActivityCompleted') THEN + RAISE EXCEPTION 'TEST FAILED [N1 execution fence]: expected execute-sql ActivityFailed for identity replacement, history=%', + (SELECT jsonb_agg(to_jsonb(history)) FROM _e2e75_sql_history AS history); + END IF; +END $$; + +CREATE TEMP TABLE _e2e75_fresh AS +SELECT * FROM dblink('e2e75_origin', $remote$ + SELECT df.start('INSERT INTO public.e2e75_effects (marker) VALUES (''fresh'')', 'e2e75-fresh') +$remote$) AS remote(local_id TEXT); +DO $$ +DECLARE + final_status TEXT; + workflow RECORD; +BEGIN + SELECT remote.status INTO final_status FROM dblink('e2e75_origin', format( + 'SELECT df.await_instance(%L, 30)', (SELECT local_id FROM _e2e75_fresh))) AS remote(status TEXT); + IF final_status IS DISTINCT FROM 'completed' THEN + RAISE EXCEPTION 'TEST FAILED [fresh satellite]: status = %', final_status; + END IF; + final_status := df.await_instance((SELECT local_id FROM _e2e75_unfenced), 30); + IF final_status IS DISTINCT FROM 'completed' THEN + RAISE EXCEPTION 'TEST FAILED [unfenced negative control]: status = %', final_status; + END IF; + FOR workflow IN SELECT * FROM _e2e75_control ORDER BY marker LOOP + final_status := df.await_instance(workflow.local_id, 30); + IF final_status IS DISTINCT FROM 'completed' THEN + RAISE EXCEPTION 'TEST FAILED [%]: status = %', workflow.marker, final_status; + END IF; + END LOOP; +END $$; + +SELECT pg_temp.e2e75_wait(format($check$ + SELECT EXISTS (SELECT 1 FROM %I.get_instance_info(%L) + WHERE lower(status) IN ('completed', 'failed', 'cancelled')) +$check$, df.duroxide_schema(), engine_id), 'retained old engine reaches terminal status', + released_at + interval '130 seconds') FROM _e2e75_old; +DO $$ +DECLARE + old_status TEXT; +BEGIN + EXECUTE format('SELECT status FROM %I.get_instance_info($1)', df.duroxide_schema()) + INTO old_status USING (SELECT engine_id FROM _e2e75_old); + IF lower(old_status) IS DISTINCT FROM 'failed' OR + NOT EXISTS (SELECT 1 FROM _e2e75_sql_history WHERE outcome_type = 'ActivityFailed' + AND error_message = 'Origin installation removed or replaced') OR + EXISTS (SELECT 1 FROM _e2e75_sql_history WHERE outcome_type = 'ActivityCompleted') THEN + RAISE EXCEPTION 'TEST FAILED [old engine]: expected retained failure and no SQL completion, status=%', old_status; + END IF; +END $$; + +SELECT dblink_exec('e2e75_origin', $remote$ + DO $check$ + BEGIN + IF (SELECT count(*) FROM public.e2e75_effects) <> 2 OR NOT EXISTS ( + SELECT 1 FROM public.e2e75_effects WHERE marker = 'fresh' + AND database_name = current_database() AND role_name = 'df_e2e_user') OR NOT EXISTS ( + SELECT 1 FROM public.e2e75_effects WHERE marker = 'unfenced-old' + AND database_name = current_database() AND role_name = 'df_e2e_user') THEN + RAISE EXCEPTION 'TEST FAILED [replacement effects]: expected fresh and unfenced-old only; identity-bound old SQL must never write'; + END IF; + END $check$; +$remote$); +DO $$ +BEGIN + IF (SELECT count(*) FROM public.e2e75_effects) <> 1 OR + EXISTS (SELECT 1 FROM _e2e75_control AS workflow WHERE NOT EXISTS ( + SELECT 1 FROM public.e2e75_effects AS effects WHERE effects.marker = workflow.marker + AND effects.database_name = current_database() AND effects.role_name = 'df_e2e_user')) OR + (SELECT count(*) FROM _e2e75_epoch) <> 1 OR + (SELECT epoch_id FROM df._worker_epoch) IS DISTINCT FROM (SELECT epoch_id FROM _e2e75_epoch) THEN + RAISE EXCEPTION 'TEST FAILED [control effects]: expected committed control marker and unchanged runtime epoch'; + END IF; +END $$; + +SELECT dblink_disconnect('e2e75_origin'); +DROP DATABASE _e2e75_origin WITH (FORCE); +BEGIN; +DELETE FROM df.nodes WHERE instance_id IN ( + SELECT local_id FROM _e2e75_control UNION ALL SELECT local_id FROM _e2e75_unfenced); +DELETE FROM df.instances WHERE id IN ( + SELECT local_id FROM _e2e75_control UNION ALL SELECT local_id FROM _e2e75_unfenced); +COMMIT; +DROP TABLE public.e2e75_effects; +DROP VIEW _e2e75_sql_history, _e2e75_guards, _e2e75_waiters; +DROP TABLE _e2e75_control, _e2e75_unfenced, _e2e75_old, _e2e75_relations, _e2e75_epoch, _e2e75_replacement, _e2e75_fresh; +DROP FUNCTION pg_temp.e2e75_wait(TEXT, TEXT, TIMESTAMPTZ); + +SELECT 'TEST PASSED: multi-database force drop execution fence' AS result; \ No newline at end of file