Upgrade to duroxide 0.1.26 and duroxide-pg-opt 0.1.23

Author: pinodeca

Cargo.lock

@@ -32,7 +32,7 @@ serde_json = "1.0"
 uuid = { version = "1.0", features = ["v4", "serde"] }
 
 # duroxide integration
-duroxide = "=0.1.20"
+duroxide = "=0.1.26"
 duroxide-pg-opt = { path = "duroxide-pg-opt", package = "duroxide-pg-opt" }
 tokio = { version = "1", features = ["rt-multi-thread", "sync", "time"] }
 

Cargo.toml

@@ -0,0 +1,201 @@
+# Plan: Update to duroxide 0.1.26 + duroxide-pg-opt 4a6bf6b
+
+## Status: Preparatory
+
+This plan documents the work required to move pg_durable to duroxide 0.1.26 and
+duroxide-pg-opt commit `4a6bf6b`.
+
+**Do not merge the implementation until there is a new duroxide-pg-opt release tag
+(expected: v0.1.24) that includes the `MigrationPolicy` commit `4a6bf6b`.** At that
+point this plan will likely need an update to account for any additional changes to
+duroxide-pg-opt between now and that tag, and possibly additional duroxide versions
+as well.
+
+## Current vs. Target
+
+| Dependency | Current | Target |
+|---|---|---|
+| `duroxide` (crates.io) | `=0.1.20` | `=0.1.26` |
+| `duroxide-pg-opt` (submodule) | `8ce65bd` (topic branch, based off ~v0.1.18) | `4a6bf6b` (origin/main, one commit past v0.1.23) |
+
+The current submodule commit (`8ce65bd`) is the `MigrationPolicy` feature branch
+that adds `MigrationPolicy::VerifyOnly`. The target commit (`4a6bf6b`) is the
+mainline commit on `origin/main` that merged the same feature — so `VerifyOnly`
+is already present in pg_durable and nothing about that support changes.
+
+**Note (commit 397fb93):** pg_durable no longer checks in duroxide migration files.
+The BGW now applies all duroxide migrations at startup via `MigrationPolicy::ApplyAll`;
+the backend connection uses `VerifyOnly`. `sql/duroxide_install.sql`,
+`sql/duroxide_upstream/`, and the scripts `gen-duroxide-install-sql.sh` and
+`verify-duroxide-migrations.sh` were deleted in that commit. Implementation steps
+that previously synced those files are therefore eliminated (see checklist below).
+
+## What Changed in duroxide (0.1.20 → 0.1.26)
+
+### 0.1.21 — Orphan queue message drop (no impact)
+
+Drops `QueueMessage` work items that arrive before orchestration starts. Internal
+provider-level fix; no pg_durable code changes needed.
+
+### 0.1.22 — Activity Tags
+
+New `TagFilter` enum, `RuntimeOptions.worker_tag_filter`, and
+`DurableFuture::with_tag()`. The `Provider::fetch_work_item()` signature gains a
+`tag_filter: &TagFilter` parameter — a breaking change for provider implementors,
+fully absorbed by duroxide-pg-opt 0.1.20+. The default is `TagFilter::DefaultOnly`
+(process only untagged activities), so all existing `schedule_activity` calls are
+completely unaffected. No pg_durable source changes required.
+
+### 0.1.23 — Bug fixes and docs only (no impact)
+
+### 0.1.24 — Durable KV Store
+
+New `ctx.set_value()` / `ctx.get_value()` orchestration API and a new
+`Provider::get_kv_value()` trait method. pg_durable does not use the KV API.
+Provider-level changes absorbed by duroxide-pg-opt 0.1.21+.
+
+### 0.1.25 — KV API Rename + Bulk Reads (no pg_durable impact)
+
+`ctx.set_value()` → `ctx.set_kv_value()` etc. New required Provider method
+`get_kv_all_values()`. `OrchestrationItem.kv_snapshot` type changes to
+`HashMap<String, KvEntry>`. All absorbed by duroxide-pg-opt.
+
+### 0.1.26 — KV Delta Table (fixes RMW replay poisoning)
+
+KV mutations during the current execution now stage to a `kv_delta` table; only
+merged to `kv_store` at execution boundaries (Completed/CAN/Failed). Fixes
+nondeterminism errors in read-modify-write patterns. Provider-and-migration-only
+change, fully absorbed by duroxide-pg-opt.
+
+**Net impact on pg_durable Rust source code: zero required changes.**
+
+## What Changed in duroxide-pg-opt (8ce65bd → 4a6bf6b)
+
+The target includes duroxide-pg-opt versions v0.1.19–v0.1.23 plus the
+`MigrationPolicy` PR merged on top.
+
+### New migrations (all additive/safe for rolling upgrades)
+
+| Migration | Feature | Schema Changes |
+|---|---|---|
+| 0006 | Activity tag routing | `worker_queue.tag TEXT` column + index; updated `enqueue_worker_work` and `fetch_work_item` SPs |
+| 0007 | Retry on fetch contention | `fetch_orchestration_item` SP body change only (no schema change) |
+| 0008 | KV store | New `kv_store` table; updated `fetch_orchestration_item`, `ack_orchestration_item`, deletion/pruning SPs |
+| 0009 | KV timestamps | `kv_store.last_updated_at_ms BIGINT` column; updated KV materialization SPs |
+| 0010 | KV delta table | New `kv_delta` table; two-table KV write model; delta→store merge on terminal transition |
+
+### API stability
+
+All pg_durable call sites against duroxide-pg-opt remain valid:
+
+- `PostgresProvider::new_with_config(url, config)` — unchanged
+- `ProviderConfig { schema_name, migration_policy, long_poll }` — unchanged
+- `MigrationPolicy::ApplyAll` (BGW) and `MigrationPolicy::VerifyOnly` (backend) — both present and in use
+- **New at `4a6bf6b`**: `schema_name` values are now validated against
+  `[A-Za-z_][A-Za-z0-9_]*` on construction. `"duroxide"` passes this check.
+
+## Implementation Checklist
+
+### 1. Update submodule pointer
+
+```bash
+cd duroxide-pg-opt
+git fetch origin
+git checkout 4a6bf6b
+cd ..
+```
+
+### 2. Update `Cargo.toml`
+
+```toml
+# Change:
+duroxide = "=0.1.20"
+# To:
+duroxide = "=0.1.26"
+```
+
+~~### 3. Sync migrations to `sql/duroxide_upstream/`~~ **Eliminated by commit 397fb93.**
+
+~~### 4. Regenerate install SQL~~ **Eliminated by commit 397fb93.**
+
+Because the BGW now applies all duroxide migrations via `MigrationPolicy::ApplyAll`,
+pg_durable no longer needs to copy migration files or maintain `sql/duroxide_install.sql`.
+The extension SQL only creates an empty `duroxide` schema; the BGW populates it at runtime.
+
+### 3. Update the pg_durable upgrade script
+
+v0.2.0 has not yet shipped, so edit the existing `sql/pg_durable--0.1.1--0.2.0.sql`
+upgrade script. **Do not add the duroxide migrations to the upgrade script** — the BGW
+will apply any missing migrations (0006–0010) via `ApplyAll` at startup.
+The only df-schema DDL needed is the `df.is_ready()` function (already added by
+commit 397fb93).
+
+### 4. Update `docs/upgrade-testing.md`
+
+Add a "Version-Specific Changes" entry for the new version covering:
+
+- **Scenario A**: The `df` schema equivalent contract holds; the `duroxide` schema
+  is explicitly excluded from snapshot diffs because fresh installs start with an
+  empty `duroxide` schema (BGW fills it in at runtime) while upgrades carry forward
+  the fully-populated schema from v0.1.1. This divergence is expected and documented.
+- **Scenario B1 (binary backward compat)**: The BGW uses `MigrationPolicy::ApplyAll`.
+  A database with only migrations 0001–0005 is handled gracefully — the BGW applies
+  the missing migrations at startup rather than refusing to initialize.
+- **Scenario B2**: No new pg_durable DDL in this release (all new tables are
+  duroxide-owned). Existing `df.*` data survives upgrade unchanged.
+
+(Note: the BGW-applies-migrations entry was already added to `docs/upgrade-testing.md`
+by commit 397fb93; this step is to document any remaining version-specific notes.)
+
+### 5. Run `cargo update`
+
+```bash
+cargo update
+```
+
+This refreshes all transitive dependencies, not just duroxide and duroxide-pg-opt.
+
+### 6. Build and test
+
+```bash
+cargo build --features pg17
+cargo clippy --features pg17
+./scripts/test-unit.sh
+./scripts/test-e2e-local.sh
+./scripts/test-upgrade.sh
+```
+
+## Upgrade & Migration Impact
+
+### B1 — Binary Backward Compatibility
+
+The BGW uses `MigrationPolicy::ApplyAll`, so it will apply any missing duroxide
+migrations (0006–0010) at startup rather than refusing to initialize. A database
+that still has only migrations 0001–0005 is safe: the BGW detects the gap and
+applies the remaining migrations in order. The backend uses `MigrationPolicy::VerifyOnly`
+and will wait (via `df.is_ready()` polling) until the BGW has completed all
+migrations before accepting work.
+
+### B2 — Data Compatibility After Upgrade
+
+All five new migrations are additive (new tables, new columns with defaults or
+nullable). Existing `df.vars`, `df.nodes`, `df.instances`, and `df.graphs` data
+is untouched. No data migration is required.
+
+## Opportunities for pg_durable Improvements
+
+These are not required for this update but become available once the dependency
+versions are bumped. Consider as follow-on work:
+
+1. **Activity Tags** (duroxide 0.1.22): Expose a `tag` parameter on `df.sql()`,
+   `df.http()`, etc. to route activities to specialized worker pools. Workers
+   would declare their `TagFilter` via a GUC or `pg_durable.worker_tag` setting.
+
+2. **Durable KV Store** (duroxide 0.1.24): Expose `df.kv_set(key, value)` and
+   `df.kv_get(key)` as first-class DSL nodes, allowing orchestrations to accumulate
+   state durably without a side table pattern. Useful for loop counter and
+   accumulator patterns.
+
+3. **Custom Status** (duroxide 0.1.19): Enrich `df.status()` to surface a custom
+   status string published from within an orchestration, enabling richer progress
+   reporting beyond the Pending/Running/Completed/Failed states.

docs/plan-duroxide-26-update.md

@@ -209,3 +209,17 @@ what the upgrade script handles, and any backward compatibility considerations.
 - **Scenario B1 considerations:** Readiness checking is internal to the Rust binary — no SQL function needs to be registered. The new `.so` continues to work against v0.1.1 schemas that have not run `ALTER EXTENSION UPDATE`.
 - **Scenario B2 considerations:** The BGW readiness check (`wait_for_ready()`) is called in both B1 and B2 scenarios to ensure the BGW has applied all pending duroxide migrations before exercising the extension.
 - **Current status on this branch:** Implemented. BGW now uses `MigrationPolicy::ApplyAll`, verifies duroxide schema extension ownership before applying, and writes `duroxide._worker_ready` after initialization completes.
+
+#### Bump to duroxide 0.1.26 + duroxide-pg-opt 4a6bf6b (migrations 0006–0010)
+- **DDL change (df schema):** None. All new schema objects are in the `duroxide` schema and are applied at runtime by the BGW.
+- **DDL change (duroxide schema):** Five new migrations applied by BGW at startup:
+  - 0006: `worker_queue.tag TEXT` column + index; updated `enqueue_worker_work` and `fetch_work_item` SPs
+  - 0007: `fetch_orchestration_item` SP body change only (no schema change)
+  - 0008: new `kv_store` table; updated `fetch_orchestration_item`, `ack_orchestration_item`, deletion/pruning SPs
+  - 0009: `kv_store.last_updated_at_ms BIGINT` column; updated KV materialization SPs
+  - 0010: new `kv_delta` table; two-table KV write model; delta→store merge on terminal transition
+- **Scenario A considerations:** The `df` schema equivalence contract is unchanged. The `duroxide` schema is excluded from snapshot diffs — fresh installs start with an empty `duroxide` schema (BGW fills it in at runtime) while upgrades carry forward the fully-populated schema from v0.1.1. This is expected and acceptable.
+- **Scenario B1 considerations:** The BGW uses `MigrationPolicy::ApplyAll`. A database that has only migrations 0001–0005 is handled gracefully: the BGW detects the gap and applies 0006–0010 at startup. No manual intervention is needed.
+- **Scenario B2 considerations:** All five new migrations are additive (new tables and columns with defaults or nullable). Existing `df.vars`, `df.nodes`, `df.instances`, and `df.graphs` data is untouched.
+- **Current status:** Implemented — submodule at `4a6bf6b`, `Cargo.toml` pinned to `duroxide = "=0.1.26"`.
+>>>>>>> dc6ab46 (Upgrade to duroxide 0.1.26 and duroxide-pg-opt 0.1.23)