feat(ocrvs-10450): legacy user migration #11048

Zangetsu101 · 2025-11-17T09:19:47Z

Introduce mongo foreign data wrapper to migrate legacy mongo data
- This required a migration to be run as the "SUPERUSER" so a separate SUPERUSER postgres migration path was introduced
Migrate the legacy users to postgres via a feature flag migrate-legacy-users
Refactor the tRPC context to allow defining public procedures that don't require authentication. The v2 seed data endpoint needs this.
Remove legacy mongo migration status outputs and skip typecheck which reduced the migration service startup time by 66%.
The postgres migration files now get restored to their original state (i.e. without the environment variables being replaced) regardless of the migration passing or not

Zangetsu101 · 2025-12-08T07:13:59Z

Could this be elaborated? Sounds unsafe without context

Basically this would allow us to create endpoints that don't require auth only IF we want. Currently we do not have any yet but after the legacy users are removed, we will be deprecating the data-seeder package and define a public endpoint in events service that will seed data.

ocrvs-bot · 2025-12-08T13:45:14Z

Your environment is deployed to https://ocrvs-10450.e2e-k8s.opencrvs.dev

Countryconfig will be the entire owner of this service.

Zangetsu101 · 2025-12-08T14:35:06Z

@makelicious I've put the legacy user migration under the migrate-legacy-users flag and we are running it as a separate service from countryconfig: https://github.com/opencrvs/opencrvs-countryconfig/pull/1172/files#diff-ba8e76e28deb79c9af80270674a0f04af642ba68d7b1c608ed5790db9cc6f4c8R949

This allows us to easily switch back to the official postgres image once the migrations have run successfully.

Zangetsu101 · 2025-11-18T07:00:15Z

packages/events/src/router/trpc.ts

+})
+
+/** @knipignore */
+export const publicProcedure = t.procedure


Initially I planned on creating a public endpoint to run the migration but afterwards moved the migration to the "migration" package. Still retaining the public procedure as the seed endpoint would potentially need it after the refactor.

Zangetsu101 · 2025-12-05T12:50:11Z

packages/migration/src/migrations/events/1764251733685_create-users-table.sql

@@ -0,0 +1,43 @@
+-- Up Migration
+CREATE TYPE status_type AS ENUM(


Yea I've been there too. Let's enforce it with zod schema then

Zangetsu101 · 2025-12-05T12:51:27Z

packages/migration/src/migrations/events/1764251733685_create-users-table.sql

+);
+
+CREATE TABLE users(
+  id uuid PRIMARY KEY DEFAULT gen_random_uuid(),


Thanks, kinda forgot to add it in

Zangetsu101 · 2025-12-05T12:59:13Z

packages/events/src/tests/global-setup.ts

 async function setupPostgresServer() {
-  return new PostgreSqlContainer('postgres:17.6')
+  return new PostgreSqlContainer(
+    'docker.io/chumaky/postgres_mongo_fdw:17.6_fdw5.5.2'


We mainly need it for our legacy data migration on 2.0. Perhaps we can switch back to official images in 2.1 and remove these mongo related migration files. Do you see any constraints on going with that route?

Zangetsu101 · 2025-12-05T13:02:26Z

packages/migration/src/migrations/migrate-legacy-users/1763101029351_create-mongo-server.sql

Actually yes, they are dependent on the timestamps and I've also had to change the timestamps if newer migrations were merged into develop first. But that only applies if all the migrations are tracked using single migration table. I've created a separate one for the superuser ones:

opencrvs-core/packages/migration/run-migrations.sh

Line 75 in 666d130

--migrations-table="$migrations_table"

Zangetsu101 · 2025-12-09T06:49:29Z

.github/workflows/build-images-from-branch.yml

formatting changes, can be ignored

Zangetsu101 · 2025-12-09T06:53:13Z

packages/migration/src/migrations/migrate-legacy-users/1764251733685_create-users-table.sql

This is basically a duplicate of the one defined in the events migration, to ensure the users table exists no matter which migrations run first, events or migrate-legacy-users.

Zangetsu101 · 2025-12-09T06:53:41Z

packages/migration/src/migrations/migrate-legacy-users/1764251733685_create-users-table.sql

+
+GRANT SELECT, INSERT, UPDATE, DELETE ON users TO ${EVENTS_DB_USER};
+
+ALTER TABLE users OWNER TO ${EVENTS_MIGRATION_USER};


Changing owner to the migration user as this runs as SUPERUSER

makelicious

Looks good! You might get better review regarding .sh changes from someone else.

This comment has been minimized.

Sign in to view

Zangetsu101 force-pushed the ocrvs-10450 branch from 92eaf05 to 639c890 Compare November 18, 2025 11:02

Zangetsu101 and others added 22 commits November 25, 2025 14:08

feat: users table creation

db72918

feat: generate user types

e0f32d3

fix: make officeId mandatory

c991d4f

feat: migration endpoint

a633369

style: format file

eb12b18

feat: public procedure that does not require auth

b0d6fd6

chore: make migrateLegacyUsers public

eefb5d0

fix: remove extra whitespace

9125901

chore: add down migration for users table

8643685

feat: mongo as foreign server in postgres

2312c1d

feat: postgres foreign table for legacy users

e75e51f

feat: postgres foreign table for legacy practitioners data

427b909

fix: include practitionerId in legacy users table

3d65f55

fix: column name

47e2eca

feat: legacy users migration to Postgres

399a69b

chore: remove legacy user migration endpoint

523ad3f

feat: restore original backup files in case of failure

20dfc33

chore: remove mongoose as dependency

c0b9e8e

chore: declare variable at the same time as its assignment

9e3d9cd

feat: separate out type check from build

5ee3d46

feat: setup required env variables

3a728a6

chore: suppress legacy migration status

6a8ead3

Zangetsu101 force-pushed the ocrvs-10450 branch from bfef61b to 6a8ead3 Compare November 25, 2025 08:11

chore: rename procedures used in custom actions

1833dca

Zangetsu101 changed the title ~~Ocrvs 10450~~ feat(ocrvs-10450): legacy user migration Nov 25, 2025

Zangetsu101 added 2 commits November 25, 2025 14:54

chore: add commons as dependency

c67275e

test: update expected UNAUTHORIZED error format

b1462be

Zangetsu101 marked this pull request as ready for review November 25, 2025 11:08

Zangetsu101 added 11 commits December 5, 2025 19:40

Merge branch 'develop' into ocrvs-10450

484bd01

chore: create user_credentials table separately

e9ad529

chore: remove status_type enum

748c50b

feat: migrate legacy users with a feature flag

5c7ab5d

feat: revert back to official postgres image

2cf8347

feat: add legacy-user-migration service

f808b98

fix: add missing NOT to create users table

6426609

fix: change ownership during legacy user migration

26655cd

fix: change status column type to text

80a3598

fix: create mapping for the current user

39d5720

fix: EVENTS_DB_USER required in both migrations

8bde079

Zangetsu101 added 5 commits December 8, 2025 14:01

Merge branch 'develop' into ocrvs-10450

d51fddf

fix: exclude legacy-user-migration from build workflow

9a8ec21

feat: include timestamps in user records

e2e0025

feat: migrate user credentials to Postgres

4feb68d

Merge branch 'develop' into ocrvs-10450

6c4c8bb

Zangetsu101 added 2 commits December 8, 2025 19:50

feat: generate new schema and types

f1a9f92

feat: remove legacy-user-migration service

9af6a9b

Countryconfig will be the entire owner of this service.

Zangetsu101 requested a review from makelicious December 8, 2025 14:33

Zangetsu101 mentioned this pull request Dec 9, 2025

chore: revert to official postgres opencrvs/infrastructure#207

Merged

chore: remove unneeded ignore

1f62ea2

Zangetsu101 commented Dec 9, 2025

View reviewed changes

Merge branch 'develop' into ocrvs-10450

ff3009f

naftis self-requested a review December 9, 2025 09:10

makelicious approved these changes Dec 9, 2025

View reviewed changes

Zangetsu101 merged commit 755e62f into develop Dec 10, 2025
123 of 124 checks passed

Zangetsu101 deleted the ocrvs-10450 branch December 10, 2025 09:40

		@@ -0,0 +1,43 @@
		-- Up Migration
		CREATE TYPE status_type AS ENUM(


		GRANT SELECT, INSERT, UPDATE, DELETE ON users TO ${EVENTS_DB_USER};

		ALTER TABLE users OWNER TO ${EVENTS_MIGRATION_USER};

feat(ocrvs-10450): legacy user migration #11048

feat(ocrvs-10450): legacy user migration #11048

Conversation

Zangetsu101 commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

Zangetsu101 commented Dec 8, 2025

Uh oh!

ocrvs-bot commented Dec 8, 2025

Uh oh!

Zangetsu101 commented Dec 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

makelicious left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Zangetsu101 commented Nov 17, 2025 •

edited

Loading