add support for hyperdisk nodes

[nicu-da]

include support for heap dumps pvcs and the migration pvs

in a follow up PR i will add support to migrate the postgres and cometbft pvcs

we basically can deploy one new pool with hyperdisk, and if we enable the flag that hyperdisk is supported then we alter the node selectors to deploy everything to the new nodes except postgres and cometbft

Pull Request Checklist

Cluster Testing

• If a cluster test is required, comment /cluster_test on this PR to request it, and ping someone with access to the DA-internal system to approve it.
• If a hard-migration test is required (from the latest release), comment /hdm_test on this PR to request it, and ping someone with access to the DA-internal system to approve it.

PR Guidelines

• Include any change that might be observable by our partners or affect their deployment in the release notes.
• Specify fixed issues with Fixes #n, and mention issues worked on using #n
• Include a screenshot for frontend-related PRs - see README or use your favorite screenshot tool

Merge Guidelines

• Make the git commit message look sensible when squash-merging on GitHub (most likely: just copy your PR description).

[nicu-da]

/cluster_test

[github-actions]

Deploy cluster test triggered for Commit 82e3c1469456cc60e19455d3cabf2490e5454b4b in , please contact a Contributor to approve it in CircleCI: https://app.circleci.com/pipelines/github/DACH-NY/canton-network-internal/48376

[nicu-da]

green cluster build https://app.circleci.com/pipelines/github/DACH-NY/canton-network-internal/48458/workflows/98318219-8946-4c5f-a3af-75ee681bc787

[martinflorian-da]

Slightly offtopic: what is our plan again for the observability PVCs? We just move the ones we care about manually, with a short downtime?

[nicu-da]

that's the infra pool, main issue is the prometheus pvc, will tackle that as a last step. will have to check what support do the helm charts have

[martinflorian-da]

/cluster_test

Without hyperdisk, correct? So this is signal that we don't break anything.

[martinflorian-da]

Looks reasonable to me, thanks for fighting this fight!

[github-actions]

Deploy cluster test triggered for Commit 80486ddc2dd67ef5b3130c821a2ccae540e8c761 in , please contact a Contributor to approve it in CircleCI: https://app.circleci.com/pipelines/github/DACH-NY/canton-network-internal/48476

[nicu-da]

/cluster_test

Without hyperdisk, correct? So this is signal that we don't break anything.

Exactly

[github-actions]

Deploy cluster test triggered for Commit 80486ddc2dd67ef5b3130c821a2ccae540e8c761 in , please contact a Contributor to approve it in CircleCI: https://app.circleci.com/pipelines/github/DACH-NY/canton-network-internal/48477