Skip to content

add support for migrating postgres pvcs to hyperdisk #3653

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nicu-da merged 2 commits into from
Jan 22, 2026
+145 −40
Merged

add support for migrating postgres pvcs to hyperdisk #3653

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e9a620f
add support for migrating postgres pvcs to hyperdisk
nicu-da Jan 22, 2026
a83cf86
Update cluster/pulumi/common/src/postgres.ts
nicu-da Jan 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 30 additions & 30 deletions cluster/expected/multi-validator/expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4877,12 +4877,7 @@
}
}
},
"db": {
"maxConnections": 1000,
"volumeSize": "100Gi"
},
"imageRepo": "us-central1-docker.pkg.dev/da-cn-shared/ghcr/digital-asset/decentralized-canton-sync-dev/docker",
"nonHyperdiskAppsAffinityAndTolerations": {
"appsAffinityAndTolerations": {
"affinity": {
"nodeAffinity": {
"requiredDuringSchedulingIgnoredDuringExecution": {
Expand Down Expand Up @@ -4914,6 +4909,11 @@
}
]
},
"db": {
"maxConnections": 1000,
"volumeSize": "100Gi"
},
"imageRepo": "us-central1-docker.pkg.dev/da-cn-shared/ghcr/digital-asset/decentralized-canton-sync-dev/docker",
"persistence": {
"secretName": "postgres-0-secret"
},
Expand Down Expand Up @@ -4973,12 +4973,7 @@
}
}
},
"db": {
"maxConnections": 1000,
"volumeSize": "100Gi"
},
"imageRepo": "us-central1-docker.pkg.dev/da-cn-shared/ghcr/digital-asset/decentralized-canton-sync-dev/docker",
"nonHyperdiskAppsAffinityAndTolerations": {
"appsAffinityAndTolerations": {
"affinity": {
"nodeAffinity": {
"requiredDuringSchedulingIgnoredDuringExecution": {
Expand Down Expand Up @@ -5010,6 +5005,11 @@
}
]
},
"db": {
"maxConnections": 1000,
"volumeSize": "100Gi"
},
"imageRepo": "us-central1-docker.pkg.dev/da-cn-shared/ghcr/digital-asset/decentralized-canton-sync-dev/docker",
"persistence": {
"secretName": "postgres-1-secret"
},
Expand Down Expand Up @@ -5069,12 +5069,7 @@
}
}
},
"db": {
"maxConnections": 1000,
"volumeSize": "100Gi"
},
"imageRepo": "us-central1-docker.pkg.dev/da-cn-shared/ghcr/digital-asset/decentralized-canton-sync-dev/docker",
"nonHyperdiskAppsAffinityAndTolerations": {
"appsAffinityAndTolerations": {
"affinity": {
"nodeAffinity": {
"requiredDuringSchedulingIgnoredDuringExecution": {
Expand Down Expand Up @@ -5106,6 +5101,11 @@
}
]
},
"db": {
"maxConnections": 1000,
"volumeSize": "100Gi"
},
"imageRepo": "us-central1-docker.pkg.dev/da-cn-shared/ghcr/digital-asset/decentralized-canton-sync-dev/docker",
"persistence": {
"secretName": "postgres-2-secret"
},
Expand Down Expand Up @@ -5165,12 +5165,7 @@
}
}
},
"db": {
"maxConnections": 1000,
"volumeSize": "100Gi"
},
"imageRepo": "us-central1-docker.pkg.dev/da-cn-shared/ghcr/digital-asset/decentralized-canton-sync-dev/docker",
"nonHyperdiskAppsAffinityAndTolerations": {
"appsAffinityAndTolerations": {
"affinity": {
"nodeAffinity": {
"requiredDuringSchedulingIgnoredDuringExecution": {
Expand Down Expand Up @@ -5202,6 +5197,11 @@
}
]
},
"db": {
"maxConnections": 1000,
"volumeSize": "100Gi"
},
"imageRepo": "us-central1-docker.pkg.dev/da-cn-shared/ghcr/digital-asset/decentralized-canton-sync-dev/docker",
"persistence": {
"secretName": "postgres-3-secret"
},
Expand Down Expand Up @@ -5261,12 +5261,7 @@
}
}
},
"db": {
"maxConnections": 1000,
"volumeSize": "100Gi"
},
"imageRepo": "us-central1-docker.pkg.dev/da-cn-shared/ghcr/digital-asset/decentralized-canton-sync-dev/docker",
"nonHyperdiskAppsAffinityAndTolerations": {
"appsAffinityAndTolerations": {
"affinity": {
"nodeAffinity": {
"requiredDuringSchedulingIgnoredDuringExecution": {
Expand Down Expand Up @@ -5298,6 +5293,11 @@
}
]
},
"db": {
"maxConnections": 1000,
"volumeSize": "100Gi"
},
"imageRepo": "us-central1-docker.pkg.dev/da-cn-shared/ghcr/digital-asset/decentralized-canton-sync-dev/docker",
"persistence": {
"secretName": "postgres-4-secret"
},
Expand Down
4 changes: 4 additions & 0 deletions cluster/helm/splice-postgres/templates/postgres.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,10 @@ spec:
storage: {{ $.Values.db.volumeSize }}
storageClassName: {{ $.Values.db.volumeStorageClass }}
volumeMode: Filesystem
{{- with .Values.db.dataSource }}
dataSource:
{{- toYaml . | nindent 8 }}
Copy link
Contributor

@martinflorian-da martinflorian-da Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you are a brave man... I would have done a helm test here just to be sure I counted the whitespaces right (I think I'd still soft recommend it)

Copy link
Contributor Author

@nicu-da nicu-da Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a full manual test to migrate a scratch

{{- end }}
---
apiVersion: v1
kind: Service
Expand Down
1 change: 1 addition & 0 deletions cluster/pulumi/common/src/config/configSchema.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ const PulumiProjectConfigSchema = z.object({
interAppsDependencies: z.boolean(),
cloudSql: CloudSqlConfigSchema,
allowDowngrade: z.boolean(),
replacePostgresStatefulSetOnChanges: z.boolean().default(false),
});
export type PulumiProjectConfig = z.infer<typeof PulumiProjectConfigSchema>;
export const ConfigSchema = z.object({
Expand Down
55 changes: 51 additions & 4 deletions cluster/pulumi/common/src/postgres.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,20 @@ import * as gcp from '@pulumi/gcp';
import * as pulumi from '@pulumi/pulumi';
import * as random from '@pulumi/random';
import * as _ from 'lodash';
import { CustomResource } from '@pulumi/kubernetes/apiextensions';
import { Resource } from '@pulumi/pulumi';

import { CnChartVersion } from './artifacts';
import { clusterSmallDisk, CloudSqlConfig, config } from './config';
import { spliceConfig } from './config/config';
import { hyperdiskSupportConfig } from './config/hyperdiskSupportConfig';
import {
appsAffinityAndTolerations,
infraAffinityAndTolerations,
installSpliceHelmChart,
nonHyperdiskAppsAffinityAndTolerations,
} from './helm';
import { installPostgresPasswordSecret } from './secrets';
import { standardStorageClassName } from './storage/storageClass';
import { ChartValues, CLUSTER_BASENAME, ExactNamespace, GCP_ZONE } from './utils';

const project = gcp.organizations.getProjectOutput({});
Expand Down Expand Up @@ -224,6 +227,35 @@ export class SplicePostgres extends pulumi.ComponentResource implements Postgres

// an initial database named cantonnet is created automatically (configured in the Helm chart).
const smallDiskSize = clusterSmallDisk ? '240Gi' : undefined;
const supportsHyperdisk =
hyperdiskSupportConfig.hyperdiskSupport.enabled && !useInfraAffinityAndTolerations;
let hyperdiskMigrationValues = {};
Copy link
Contributor

@martinflorian-da martinflorian-da Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason not do const hyperdiskMigrationValues = xyz ? { ... } : { } like we usually do?

Copy link
Contributor Author

@nicu-da nicu-da Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have to create the snapshot in the if

Copy link
Contributor

@martinflorian-da martinflorian-da Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah I see; this sounds like a reason then...

want to add a brief comment? (nit)

Copy link
Contributor Author

@nicu-da nicu-da Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see the value of the comment tbh, it would comment on the syntax of typescript not add any extra details about the why

Copy link
Contributor

@martinflorian-da martinflorian-da Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, why can't you write this in functional style again?

Would something like this work?

const hyperdiskMigrationValues = (xyz ? (() => { new xyz; return {abc}) : (() => {}))()

(all of this is nit; I just stumbled about the inconsistency so figured other readers might too; we very rarely use mutable variables)

Copy link
Contributor Author

@nicu-da nicu-da Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do use mutable variables in tyepscript when needed.
I find that the mutable state with the if is much cleaner to read compared to an anonymous function that gets called in the same line. It took me a few reads to figure out the nested complexity there.

if (supportsHyperdisk && hyperdiskSupportConfig.hyperdiskSupport.migrating) {
const pvcSnapshot = new CustomResource(
`pg-data-${xns.logicalName}-${instanceName}-snapshot`,
{
apiVersion: 'snapshot.storage.k8s.io/v1',
kind: 'VolumeSnapshot',
metadata: {
name: `pg-data-${instanceName}-snapshot`,
namespace: xns.logicalName,
},
spec: {
volumeSnapshotClassName: 'dev-vsc',
Copy link
Contributor

@martinflorian-da martinflorian-da Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

offtopic, just ranting: that name...

Copy link
Contributor Author

@nicu-da nicu-da Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have no idea who created that, it's in all the clusters 🤷 not managed by us

source: {
persistentVolumeClaimName: `pg-data-${instanceName}-0`,
},
},
}
);
hyperdiskMigrationValues = {
dataSource: {
kind: 'VolumeSnapshot',
name: pvcSnapshot.metadata.name,
apiGroup: 'snapshot.storage.k8s.io',
},
};
}
const pg = installSpliceHelmChart(
xns,
instanceName,
Expand All @@ -233,6 +265,13 @@ export class SplicePostgres extends pulumi.ComponentResource implements Postgres
volumeSize: overrideDbSizeFromValues
? values?.db?.volumeSize || smallDiskSize
: smallDiskSize,
...(supportsHyperdisk
? {
volumeStorageClass: standardStorageClassName,
pvcTemplateName: 'pg-data-hd',
...hyperdiskMigrationValues,
}
: {}),
},
persistence: {
secretName: this.secretName,
Expand All @@ -242,11 +281,19 @@ export class SplicePostgres extends pulumi.ComponentResource implements Postgres
{
aliases: [{ name: logicalNameAlias, type: 'kubernetes:helm.sh/v3:Release' }],
dependsOn: [passwordSecret],
...((supportsHyperdisk &&
// during the migration we first delete the stateful set, which keeps the old pvcs, and the recreate with the new pvcs
// the stateful sets are immutable so they need to be recreated to force the change of the pvcs
hyperdiskSupportConfig.hyperdiskSupport.migrating) ||
spliceConfig.pulumiProjectConfig.replacePostgresStatefulSetOnChanges
? {
replaceOnChanges: ['*'],
deleteBeforeReplace: true,
}
: {}),
},
true,
useInfraAffinityAndTolerations
? infraAffinityAndTolerations
: nonHyperdiskAppsAffinityAndTolerations
useInfraAffinityAndTolerations ? infraAffinityAndTolerations : appsAffinityAndTolerations
);
this.pg = pg;

Expand Down
65 changes: 59 additions & 6 deletions cluster/pulumi/multi-validator/src/postgres.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,18 @@ import * as pulumi from '@pulumi/pulumi';
import * as random from '@pulumi/random';
import {
activeVersion,
appsAffinityAndTolerations,
CnInput,
ExactNamespace,
installSpliceRunbookHelmChart,
installPostgresPasswordSecret,
InstalledHelmChart,
nonHyperdiskAppsAffinityAndTolerations,
installPostgresPasswordSecret,
installSpliceRunbookHelmChart,
spliceConfig,
standardStorageClassName,
} from '@lfdecentralizedtrust/splice-pulumi-common';
import { CustomResource } from '@pulumi/kubernetes/apiextensions';

import { hyperdiskSupportConfig } from '../../common/src/config/hyperdiskSupportConfig';
import { multiValidatorConfig } from './config';

export function installPostgres(
Expand All @@ -32,17 +36,66 @@ export function installPostgres(
}
const config = multiValidatorConfig!;

let hyperdiskMigrationValues = {};
if (
hyperdiskSupportConfig.hyperdiskSupport.enabled &&
hyperdiskSupportConfig.hyperdiskSupport.migrating
) {
const pvcSnapshot = new CustomResource(`pg-data-${xns.logicalName}-${name}-snapshot`, {
apiVersion: 'snapshot.storage.k8s.io/v1',
kind: 'VolumeSnapshot',
metadata: {
name: `pg-data-${name}-snapshot`,
namespace: xns.logicalName,
},
spec: {
volumeSnapshotClassName: 'dev-vsc',
source: {
persistentVolumeClaimName: `pg-data-${name}-0`,
},
},
});
hyperdiskMigrationValues = {
dataSource: {
kind: 'VolumeSnapshot',
name: pvcSnapshot.metadata.name,
apiGroup: 'snapshot.storage.k8s.io',
},
};
}
return installSpliceRunbookHelmChart(
xns,
name,
'splice-postgres',
{
persistence: { secretName },
db: { volumeSize: config.postgresPvcSize, maxConnections: 1000 },
db: {
volumeSize: config.postgresPvcSize,
maxConnections: 1000,
...(hyperdiskSupportConfig.hyperdiskSupport.enabled
? {
volumeStorageClass: standardStorageClassName,
pvcTemplateName: 'pg-data-hd',
...hyperdiskMigrationValues,
}
: {}),
},
resources: config.resources?.postgres,
nonHyperdiskAppsAffinityAndTolerations,
appsAffinityAndTolerations,
},
activeVersion,
{ dependsOn: [passwordSecret, ...dependsOn] }
{
dependsOn: [passwordSecret, ...dependsOn],
...((hyperdiskSupportConfig.hyperdiskSupport.enabled &&
// during the migration we first delete the stateful set, which keeps the old pvcs, and the recreate with the new pvcs
// the stateful sets are immutable so they need to be recreated to force the change of the pvcs
hyperdiskSupportConfig.hyperdiskSupport.migrating) ||
spliceConfig.pulumiProjectConfig.replacePostgresStatefulSetOnChanges
? {
replaceOnChanges: ['*'],
deleteBeforeReplace: true,
}
: {}),
}
);
}