Skip to content

Commit 671205e

Browse files
bjcoombsbjcoombs
authored
fix: repair Atlas migration checksums and CockroachDB compatibility (#914)
* fix: repair Atlas migration checksums and version conflicts - Regenerate atlas.sum for internal-bank-account (missing hash for 20260207000001_create_lien_table.sql added in PR #804) - Add missing atlas.sum for market-information, control-plane, forecasting, reconciliation, and reference-data services - Renumber control-plane migrations to resolve version collision: three PRs (#835, #839, #840) merged with the same version 20260209000001. Renumbered to 000001 (staff_identity), 000002 (create_manifest_versions), 000003 (manifest_versions), 000004 (manifest_apply_jobs) matching merge chronology * fix: remove PL/pgSQL trigger from market-information initial migration CockroachDB does not support PL/pgSQL in user-defined functions. The enforce_dataset_lifecycle trigger was never applied (atlas.sum was missing, blocking all migrations). Lifecycle enforcement already exists at the application layer (domain/dataset_status.go and service/dataset_service.go). CHECK constraints remain for DB-level safety. Note: position-keeping and reference-data services also have PL/pgSQL triggers that will need similar treatment on fresh CockroachDB clusters. * fix: defer partial index creation for CockroachDB compatibility CockroachDB cannot create a partial index on a column added in the same transaction (column is not yet "public"). Split the is_shared partial index from 20260119000002 into a separate migration 20260119000003 so the column is committed before the index references it. * fix: split market-information migration for CockroachDB transaction visibility CockroachDB cannot reference columns added by ALTER TABLE or create partial indexes on them within the same transaction. Split migration 20260119000002 into four parts: - 000002: ADD COLUMN + CREATE TABLE (DDL only) - 000003: CREATE INDEX on newly-added is_shared column - 000004: UPDATE seed data referencing is_shared column * fix: replace date_trunc expression indexes with plain column indexes CockroachDB does not support context-dependent operators like date_trunc() in expression indexes. Replace with plain (created_at DESC, id DESC) indexes which provide equivalent cursor ordering. The Go repository queries still use date_trunc() for second-level precision but benefit from the plain column index for ordering. * fix: remove COMMENT ON INDEX for CockroachDB compatibility CockroachDB requires table-qualified syntax for COMMENT ON INDEX (table@index) which differs from PostgreSQL's bare index name syntax. Removed COMMENT ON INDEX statements and also removed CONCURRENTLY (CockroachDB creates indexes online by default). --------- Co-authored-by: Ben Coombs <bjcoombs@users.noreply.github.com>
1 parent 16c3597 commit 671205e

14 files changed

Lines changed: 87 additions & 111 deletions

services/control-plane/migrations/20260209000001_create_manifest_versions.sql renamed to services/control-plane/migrations/20260209000002_create_manifest_versions.sql

File renamed without changes.

services/control-plane/migrations/20260209000001_manifest_versions.sql renamed to services/control-plane/migrations/20260209000003_manifest_versions.sql

File renamed without changes.

services/control-plane/migrations/20260209000002_manifest_apply_jobs.sql renamed to services/control-plane/migrations/20260209000004_manifest_apply_jobs.sql

File renamed without changes.

‎services/control-plane/migrations/atlas.sum‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
h1:KtDz8EZkrnkFKf8wp/earpPk074tUgMeod4Ah7QxYBI=
2+
20260209000001_staff_identity.sql h1:OR9Cq4l2MfrW7It8jLDt0Kzg/z1th9KJdSlZEVpOhqA=
3+
20260209000002_create_manifest_versions.sql h1:1Qzl3JK0Ah3DP0Vm/t4WsKBqXfMvNCyl2YU0pUxbYA0=
4+
20260209000003_manifest_versions.sql h1:uaOBuNtyDQd6MSQHbe0+yTtCg0Q/zcMCgXhKeMbTm8g=
5+
20260209000004_manifest_apply_jobs.sql h1:QZim1Z/9bk7/cDW5VLr+bDaP4Sgv9hFzxM/msvTYm4M=

‎services/forecasting/migrations/atlas.sum‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
h1:G6dVGofKrWaTVqX03AaL0H+U0VcpUYTX1HKLXdNoxzs=
2+
20260210000001_initial.sql h1:nkcJ9QfgP1iZPCXr5r0Go6E5+UmZKbuEY6eDIr8x508=

‎services/internal-bank-account/migrations/atlas.sum‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
1-
h1:J5gH3Yr2Od51pm2Fy651n+PQaT7HwiK7qUd9tCHJI/E=
1+
h1:E9D2JaCeSS5YvvvlwsxPss0jYsn9PzbGuvZFcjfGB5k=
22
20260112000001_initial.sql h1:DGFY67QMovT7UV5WkUEOrSZE7dLYIppV90Ix2x+qMls=
33
20260116000001_add_clearing_purpose_column.sql h1:DHT3cTeEUEWjJEysajU7QV4uy5D+7N4nuygKLsjrzn8=
44
20260116000002_backfill_clearing_purpose.sql h1:hvClf5iYdeyiKLCN3tg5l86OKAOMF7Lby+Bi+0Lyrls=
55
20260206000001_create_valuation_features.sql h1:sYLQ60lFlttH+fIvVFFFNfyBQL0ZFXb8gm0mJg7jWlE=
6+
20260207000001_create_lien_table.sql h1:WWly+7jlr0CcyWAKOqGqros+1I1MTEtteMxJPpxyRqU=

‎services/market-information/migrations/20260116000001_initial.sql‎

Lines changed: 5 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -120,67 +120,11 @@ CREATE INDEX "idx_dataset_definition_created_at" ON "dataset_definition" ("creat
120120
-- Soft-delete support: index for finding non-deleted records
121121
CREATE INDEX "idx_dataset_definition_deleted_at" ON "dataset_definition" ("deleted_at");
122122

123-
-- Trigger function to enforce dataset lifecycle rules
124-
-- Immutable fields cannot be changed once dataset is ACTIVE or DEPRECATED
125-
-- Status transitions are strictly controlled
126-
CREATE OR REPLACE FUNCTION "enforce_dataset_lifecycle"()
127-
RETURNS TRIGGER AS $$
128-
BEGIN
129-
-- Always update updated_at on any change
130-
NEW."updated_at" = NOW();
131-
132-
-- Allow all edits when status is DRAFT
133-
IF OLD."status" = 'DRAFT' THEN
134-
-- If transitioning from DRAFT to ACTIVE, set activated_at
135-
IF NEW."status" = 'ACTIVE' THEN
136-
NEW."activated_at" = NOW();
137-
-- If transitioning from DRAFT to DEPRECATED, set deprecated_at
138-
ELSIF NEW."status" = 'DEPRECATED' THEN
139-
NEW."deprecated_at" = NOW();
140-
END IF;
141-
RETURN NEW;
142-
END IF;
143-
144-
-- For ACTIVE or DEPRECATED datasets, certain fields become immutable
145-
IF OLD."status" IN ('ACTIVE', 'DEPRECATED') THEN
146-
-- Prevent changes to immutable fields (validation rules that affect data integrity)
147-
IF OLD."validation_expression" IS DISTINCT FROM NEW."validation_expression" THEN
148-
RAISE EXCEPTION 'Cannot modify validation_expression for % dataset', OLD."status";
149-
END IF;
150-
IF OLD."resolution_key_expression" IS DISTINCT FROM NEW."resolution_key_expression" THEN
151-
RAISE EXCEPTION 'Cannot modify resolution_key_expression for % dataset', OLD."status";
152-
END IF;
153-
IF OLD."error_message_expression" IS DISTINCT FROM NEW."error_message_expression" THEN
154-
RAISE EXCEPTION 'Cannot modify error_message_expression for % dataset', OLD."status";
155-
END IF;
156-
END IF;
157-
158-
-- Prevent invalid status transitions
159-
IF OLD."status" = 'ACTIVE' THEN
160-
IF NEW."status" = 'DRAFT' THEN
161-
RAISE EXCEPTION 'Cannot transition from ACTIVE back to DRAFT';
162-
END IF;
163-
-- Allow ACTIVE to DEPRECATED, set deprecated_at
164-
IF NEW."status" = 'DEPRECATED' THEN
165-
NEW."deprecated_at" = NOW();
166-
END IF;
167-
END IF;
168-
169-
IF OLD."status" = 'DEPRECATED' THEN
170-
IF NEW."status" IN ('DRAFT', 'ACTIVE') THEN
171-
RAISE EXCEPTION 'Cannot transition from DEPRECATED to %', NEW."status";
172-
END IF;
173-
END IF;
174-
175-
RETURN NEW;
176-
END;
177-
$$ LANGUAGE plpgsql;
178-
179-
-- Create trigger to enforce lifecycle rules on UPDATE
180-
CREATE TRIGGER "trg_enforce_dataset_lifecycle"
181-
BEFORE UPDATE ON "dataset_definition"
182-
FOR EACH ROW
183-
EXECUTE FUNCTION "enforce_dataset_lifecycle"();
123+
-- NOTE: Dataset lifecycle enforcement (immutable fields, status transitions,
124+
-- timestamp management) is handled at the application layer in
125+
-- domain/dataset_status.go and service/dataset_service.go.
126+
-- CockroachDB does not support PL/pgSQL triggers in user-defined functions.
127+
-- CHECK constraints above provide the database-level safety net.
184128

185129
--------------------------------------------------------------------------------
186130
-- Section 3: Market Price Observation Table

‎services/market-information/migrations/20260119000002_add_shared_dataset_support.sql‎

Lines changed: 5 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,9 @@ ALTER TABLE dataset_definition
2525
ADD CONSTRAINT chk_dataset_definition_access_level
2626
CHECK (access_level IN ('PUBLIC', 'PRIVATE', 'RESTRICTED'));
2727

28-
-- Index for finding shared datasets
29-
CREATE INDEX idx_dataset_definition_is_shared ON dataset_definition (is_shared) WHERE is_shared = TRUE;
28+
-- NOTE: Partial index on is_shared is created in a separate migration
29+
-- (20260119000003) because CockroachDB cannot create a partial index on a
30+
-- column added in the same transaction (column is not yet "public").
3031

3132
--------------------------------------------------------------------------------
3233
-- Section 2: Create tenant_data_entitlements table
@@ -67,31 +68,5 @@ CREATE INDEX idx_entitlements_expires_at
6768
ON tenant_data_entitlements(expires_at)
6869
WHERE expires_at IS NOT NULL AND is_active = TRUE;
6970

70-
--------------------------------------------------------------------------------
71-
-- Section 3: Mark ECB datasets as shared/public (forward-looking)
72-
--------------------------------------------------------------------------------
73-
74-
-- This UPDATE is idempotent and forward-looking: it will mark ECB datasets as shared
75-
-- when they are created by the ECB adapter worker. Until then, this safely matches
76-
-- zero rows since seed data uses 'FX_RATE' (a generic test dataset), not ECB-specific codes.
77-
--
78-
-- When the ECB adapter creates datasets (e.g., ECB_DAILY_FX), re-running this migration
79-
-- or creating a new migration with the same logic will enable multi-tenant sharing.
80-
--
81-
-- Explicit list of ECB foreign exchange datasets to mark as shared:
82-
-- - ECB_DAILY_FX: Daily ECB foreign exchange reference rates (created by ECB adapter)
83-
--
84-
-- Using explicit IN clause rather than LIKE pattern for predictability.
85-
-- If additional ECB datasets need to be shared, add them here explicitly.
86-
UPDATE dataset_definition
87-
SET
88-
is_shared = TRUE,
89-
access_level = 'PUBLIC',
90-
updated_at = NOW(),
91-
updated_by = 'MIGRATION'
92-
WHERE code IN ('ECB_DAILY_FX')
93-
AND status = 'ACTIVE'
94-
AND deleted_at IS NULL;
95-
96-
-- Note: No need to populate tenant_data_entitlements for PUBLIC datasets.
97-
-- PUBLIC datasets skip entitlement checks and allow access to all tenants.
71+
-- NOTE: ECB dataset seed update moved to 20260119000004 because CockroachDB
72+
-- cannot reference columns added by ALTER TABLE in the same transaction.

‎services/market-information/migrations/20260119000003_add_shared_dataset_index.sql‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
-- Deferred index creation for is_shared column added in 20260119000002.
2+
-- CockroachDB requires the column to be "public" (committed in a prior
3+
-- transaction) before a partial index can reference it.
4+
5+
CREATE INDEX idx_dataset_definition_is_shared ON dataset_definition (is_shared) WHERE is_shared = TRUE;

‎services/market-information/migrations/20260119000004_seed_shared_ecb_datasets.sql‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
-- Mark ECB datasets as shared/public (deferred from 20260119000002).
2+
-- CockroachDB requires columns added by ALTER TABLE to be committed in a
3+
-- prior transaction before they can be referenced by DML statements.
4+
--
5+
-- This UPDATE is idempotent and forward-looking: it will mark ECB datasets as shared
6+
-- when they are created by the ECB adapter worker. Until then, this safely matches
7+
-- zero rows since seed data uses 'FX_RATE' (a generic test dataset), not ECB-specific codes.
8+
UPDATE dataset_definition
9+
SET
10+
is_shared = TRUE,
11+
access_level = 'PUBLIC',
12+
updated_at = NOW(),
13+
updated_by = 'MIGRATION'
14+
WHERE code IN ('ECB_DAILY_FX')
15+
AND status = 'ACTIVE'
16+
AND deleted_at IS NULL;

0 commit comments

Comments
 (0)