feat: integrate processor partition migrator in processor and embedded apps by atzoum · Pull Request #6622 · rudderlabs/rudder-server

atzoum · 2026-01-21T10:59:52Z

🔒 Scanned for secrets using gitleaks 8.30.0

Description

Integrating processor partition migrator in embedded and processor apps & introducing an integration test for the embedded scenario:

Propagating the cancel function from main to the runner so that source migrator is able to shutdown the server in case of a timeout while waiting for inflight jobs to settle.
Introducing clustertest.PartitionRoutingProxy, a simple http reverse proxy which routes requests to the proper server backend based on user id.
Introducing clustertest.PartitionMigrationExecutor, a component which is able to orchestrate one single partition migration through etcd.
Adding optional support for router's network handle to include a X-Rudder-Instance-Id header in its requests.
Panicking in jobsdb in case readerCapacity & writerCapacity are uninitialized.
Creating the partition_id index in jobsdb regardless of the numPartitions configuration option, since buffer JobsDBs are going to have numPartitions: 0, but still, we need to be able to get jobs out of them for specific partitions.
Including integration tests in ci that were previously not running.
Gathering coverage data from rudder-server binary when running as separate process.

Linear Ticket

resolves PIPE-2700

Security

The code changed/added as part of this pull request won't create any security issues with how the software is being used.

atzoum · 2026-01-21T13:05:51Z

jobsdb/jobsdb.go

-		if _, err := tx.ExecContext(ctx, fmt.Sprintf(`CREATE INDEX "idx_%[1]s_partid" ON %[1]q (partition_id)`, newDS.JobTable)); err != nil {
-			return fmt.Errorf("creating partition_id index: %w", err)
-		}
+	if _, err := tx.ExecContext(ctx, fmt.Sprintf(`CREATE INDEX "idx_%[1]s_partid" ON %[1]q (partition_id)`, newDS.JobTable)); err != nil {


Note: we are creating the index regardless of the numPartitions configuration option, since buffer JobsDBs will have numPartitions: 0, but still, we need to be able to get jobs out of them for specific partitions.
We don't want to enable numPartitions for buffer JobsDBs, since there is no need to create xx_buf_read_excluded_partitions and xx_buf_buffered_partitionstables for them or create any extra entries in the buffered_partitions_versions table.

atzoum · 2026-01-21T13:10:50Z

app/apphandlers/setup_partitionmigration.go

+	}
+
+	targetURLProvider, err := func() (func(targetNodeIndex int) (string, error), error) {
+		processorNodeHostPattern := config.GetStringVar("", "PROCESSOR_NODE_HOST_PATTERN")


Note: this is a new env variable we'll be adding in processor pods

atzoum · 2026-01-21T13:11:15Z

app/apphandlers/setup_partitionmigration.go

+	if err != nil {
+		return nil, nil, nil, nil, fmt.Errorf("getting etcd client: %w", err)
+	}
+	nodeIndex := config.GetIntVar(-1, 1, "PROCESSOR_INDEX")


Note: this is an existing env variable

atzoum · 2026-01-21T13:13:11Z

app/apphandlers/setup_partitionmigration.go

+		}
+	}
+
+	log := logger.NewLogger().Child("partitionmigrator")


Note: using the same logger across components so that it is easier to setup alerts

codecov · 2026-01-21T15:13:43Z

Codecov Report

❌ Patch coverage is 68.82883% with 173 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.68%. Comparing base (1cffea7) to head (fa8b606).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
app/apphandlers/setup_partitionmigration.go	69.49%	38 Missing and 16 partials ⚠️
app/apphandlers/processorAppHandler.go	2.00%	49 Missing ⚠️
testhelper/clustertest/migrationexecutor.go	71.73%	14 Missing and 12 partials ⚠️
testhelper/rudderserver/rudderserver.go	66.66%	8 Missing and 4 partials ⚠️
testhelper/clustertest/routingproxy.go	66.66%	8 Missing and 2 partials ⚠️
app/apphandlers/gatewayAppHandler.go	60.00%	3 Missing and 3 partials ⚠️
app/apphandlers/embeddedAppHandler.go	92.59%	2 Missing and 2 partials ⚠️
app/cluster/dynamic.go	50.00%	1 Missing and 1 partial ⚠️
app/cluster/state/etcd.go	50.00%	1 Missing and 1 partial ⚠️
app/cluster/state/static.go	0.00%	2 Missing ⚠️
... and 4 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6622      +/-   ##
==========================================
+ Coverage   79.56%   79.68%   +0.12%     
==========================================
  Files         556      562       +6     
  Lines       62176    62829     +653     
==========================================
+ Hits        49470    50068     +598     
- Misses       9783     9794      +11     
- Partials     2923     2967      +44

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…d apps 🔒 Scanned for secrets using gitleaks 8.30.0

🔒 Scanned for secrets using gitleaks 8.30.0

mihir20

LGTM

🤖 I have created a release *beep* *boop* --- ## [1.67.0-rc.1](v1.66.0...v1.67.0-rc.1) (2026-02-02) ### Features * gateway partition migrator ([#6631](#6631)) ([344c85e](344c85e)) * integrate processor partition migrator in processor and embedded apps ([#6622](#6622)) ([b5ee878](b5ee878)) * processor partition migrator watch for new migrations ([#6611](#6611)) ([8bfb2ca](8bfb2ca)) * source node partition migrator ([#6614](#6614)) ([9e8b458](9e8b458)) * target node partition migrator ([#6617](#6617)) ([1cffea7](1cffea7)) * transformation language support ([#6637](#6637)) ([c989784](c989784)) ### Bug Fixes * negative pending events are recorded for destinations after startup ([#6619](#6619)) ([1e43b02](1e43b02)) * openapi definition validation ([#6643](#6643)) ([905f2ee](905f2ee)) ### Miscellaneous * cleanup code to push image to dockerhub for rudder-server ent ([#6628](#6628)) ([cf59323](cf59323)) * sync release v1.66.0 to main branch ([#6618](#6618)) ([c0b9619](c0b9619)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- ## [1.67.0-rc.2](v1.66.0...v1.67.0-rc.2) (2026-02-02) ### Features * gateway partition migrator ([#6631](#6631)) ([344c85e](344c85e)) * integrate processor partition migrator in processor and embedded apps ([#6622](#6622)) ([b5ee878](b5ee878)) * processor partition migrator watch for new migrations ([#6611](#6611)) ([8bfb2ca](8bfb2ca)) * source node partition migrator ([#6614](#6614)) ([9e8b458](9e8b458)) * target node partition migrator ([#6617](#6617)) ([1cffea7](1cffea7)) * transformation language support ([#6637](#6637)) ([c989784](c989784)) ### Bug Fixes * negative pending events are recorded for destinations after startup ([#6619](#6619)) ([1e43b02](1e43b02)) * openapi definition validation ([#6643](#6643)) ([905f2ee](905f2ee)) * tracking plan causing a panic in case of transformer non 200 status code response ([#6653](#6653)) ([b51ca68](b51ca68)) ### Miscellaneous * cleanup code to push image to dockerhub for rudder-server ent ([#6628](#6628)) ([cf59323](cf59323)) * ecr builds rate limited ([#6657](#6657)) ([59c2edb](59c2edb)) * migrate from PAT to GitHub App token (SEC-58) ([#6641](#6641)) ([3c78db1](3c78db1)) * sync release v1.66.0 to main branch ([#6618](#6618)) ([c0b9619](c0b9619)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: rudderstack-github-actions[bot] <236995729+rudderstack-github-actions[bot]@users.noreply.github.com>

🤖 I have created a release *beep* *boop* --- ## [1.67.0](v1.66.0...v1.67.0) (2026-02-02) ### Features * gateway partition migrator ([#6631](#6631)) ([344c85e](344c85e)) * integrate processor partition migrator in processor and embedded apps ([#6622](#6622)) ([b5ee878](b5ee878)) * processor partition migrator watch for new migrations ([#6611](#6611)) ([8bfb2ca](8bfb2ca)) * source node partition migrator ([#6614](#6614)) ([9e8b458](9e8b458)) * target node partition migrator ([#6617](#6617)) ([1cffea7](1cffea7)) * transformation language support ([#6637](#6637)) ([c989784](c989784)) ### Bug Fixes * negative pending events are recorded for destinations after startup ([#6619](#6619)) ([1e43b02](1e43b02)) * openapi definition validation ([#6643](#6643)) ([905f2ee](905f2ee)) * tracking plan causing a panic in case of transformer non 200 status code response ([#6653](#6653)) ([b51ca68](b51ca68)) ### Miscellaneous * cleanup code to push image to dockerhub for rudder-server ent ([#6628](#6628)) ([cf59323](cf59323)) * ecr builds rate limited ([#6657](#6657)) ([59c2edb](59c2edb)) * migrate from PAT to GitHub App token (SEC-58) ([#6641](#6641)) ([3c78db1](3c78db1)) * sync release v1.66.0 to main branch ([#6618](#6618)) ([c0b9619](c0b9619)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: rudderstack-github-actions[bot] <236995729+rudderstack-github-actions[bot]@users.noreply.github.com>

atzoum force-pushed the feat.processorMigratorIntegration branch 2 times, most recently from 97d5ca7 to 64de959 Compare January 21, 2026 12:00

atzoum force-pushed the feat.processorTargetMigrator branch from 768aed6 to ac24846 Compare January 21, 2026 12:32

atzoum force-pushed the feat.processorMigratorIntegration branch from 64de959 to 6fc3ce2 Compare January 21, 2026 12:32

atzoum commented Jan 21, 2026

View reviewed changes

atzoum changed the title ~~[WIP] feat: integrate processor partition migrator in processor and embedde…~~ [WIP] feat: integrate processor partition migrator in processor and embedded apps Jan 21, 2026

atzoum force-pushed the feat.processorMigratorIntegration branch from d430532 to 7e8deaf Compare January 21, 2026 15:14

atzoum force-pushed the feat.processorTargetMigrator branch from ac24846 to e2ec244 Compare January 23, 2026 11:14

atzoum force-pushed the feat.processorMigratorIntegration branch 2 times, most recently from 9f9adeb to c5c0f76 Compare January 23, 2026 20:45

atzoum force-pushed the feat.processorTargetMigrator branch from e2ec244 to f6fb61b Compare January 26, 2026 07:18

atzoum force-pushed the feat.processorMigratorIntegration branch from c2da87a to 316abaa Compare January 26, 2026 07:18

Base automatically changed from feat.processorTargetMigrator to master January 26, 2026 07:45

atzoum force-pushed the feat.processorMigratorIntegration branch from 316abaa to c9ea36a Compare January 26, 2026 07:49

atzoum changed the title ~~[WIP] feat: integrate processor partition migrator in processor and embedded apps~~ feat: integrate processor partition migrator in processor and embedded apps Jan 26, 2026

feat: integrate processor partition migrator in processor and embedde…

40aa461

…d apps 🔒 Scanned for secrets using gitleaks 8.30.0

atzoum force-pushed the feat.processorMigratorIntegration branch from c9ea36a to 40aa461 Compare January 26, 2026 10:05

chore: capture profile data from integration test

f0368bc

🔒 Scanned for secrets using gitleaks 8.30.0

atzoum force-pushed the feat.processorMigratorIntegration branch from 37eeb9a to f0368bc Compare January 26, 2026 14:07

chore: other binaries

fa8b606

🔒 Scanned for secrets using gitleaks 8.30.0

atzoum marked this pull request as ready for review January 26, 2026 15:04

atzoum requested review from fracasula, itsmihir, ktgowtham and mihir20 January 26, 2026 15:04

ktgowtham approved these changes Jan 28, 2026

View reviewed changes

mihir20 approved these changes Jan 30, 2026

View reviewed changes

atzoum merged commit b5ee878 into master Jan 30, 2026
71 checks passed

atzoum deleted the feat.processorMigratorIntegration branch January 30, 2026 14:22

This was referenced Feb 2, 2026

chore: release 1.67.0 #6647

Closed

chore: prerelease 1.67.0-rc.1 #6648

Closed

chore: release 1.67.0 #6649

Closed

chore: prerelease 1.67.0-rc.1 #6650

Merged

This was referenced Feb 2, 2026

chore: release 1.67.0 #6654

Closed

chore: prerelease 1.67.0-rc.2 #6655

Merged

chore: release 1.67.0 #6658

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: integrate processor partition migrator in processor and embedded apps#6622

feat: integrate processor partition migrator in processor and embedded apps#6622
atzoum merged 3 commits intomasterfrom
feat.processorMigratorIntegration

atzoum commented Jan 21, 2026 •

edited

Loading

Uh oh!

atzoum Jan 21, 2026 •

edited

Loading

Uh oh!

atzoum Jan 21, 2026

Uh oh!

atzoum Jan 21, 2026

Uh oh!

atzoum Jan 21, 2026

Uh oh!

codecov bot commented Jan 21, 2026 •

edited

Loading

Uh oh!

mihir20 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

atzoum commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Linear Ticket

Security

Uh oh!

atzoum Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

atzoum Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

atzoum Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

atzoum Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mihir20 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

atzoum commented Jan 21, 2026 •

edited

Loading

atzoum Jan 21, 2026 •

edited

Loading

codecov bot commented Jan 21, 2026 •

edited

Loading