[Proposal] Support SingleNode Deployment

[Ray-Eldath]

Proposers

• @avamingli
• @wfnuser
• @Ray-Eldath

Proposal Status

In Progress

Abstract

This proposal adds single node deployment to CBDB. It allows user to start a zero segment (no primary-mirror pairs) but still use, well, most of functionalities of a normal CBDB cluster.

Motivation

Customers have reported that a singlenode mode may be necessary. Since GPDB isn't fully compatible with Postgres, many features and grammars have been dependent upon. In a scenario where only one node is needed but GPDB is already relied by business logic, a singlenode mode can be used.

Implementation

Previously @wfnuser already crafted a draft version of singlenode (#77) but was deemed too radical and got rejected. Currently proposed implementation is nearly identical to that one, with the exception that no new gp_role are added. The implementation will be trifold:

Code

• bool-typed GUC gp_internal_is_singlenode has been added. The GUC can only be set through postgresql.conf and shouldn't change by user manually. This GUC is to support IS_SINGLENODE() and IS_UTILITY_BUT_NOT_SINGLENODE() macros.
• gpinitsystem will modify this GUC when creating cluster, while gpstart will read it during the first phase of initialization and boot a utility coordinator-only "cluster" accordindly in the second phase.

  for those who don't know, gpstart starts the cluster in two phases. In the first phase a coordinator-only QD is booted in utility mode so that the script can retrieve all necessary info (version,gp_segment_configuration, etc.), and in the second phase the real QD is booted with dispatch mode after all segments are created and initialized using the configuration that're read in the first phase.

• In the previous version of SingleNode (by @wfnuser), the QD is running under a new gp_role, i.e. GP_ROLE_SINGLENODE. The new role has been removed by taking the following global replacement:
  • IS_UTILITY_OR_SINGLENODE(Gp_role) -> Gp_role == GP_ROLE_UTILITY
  • !Gp_role == GP_ROLE_UTILITY -> Gp_role != GP_ROLE_UTILITY
  • Gp_role != GP_ROLE_SINGLENODE -> !IS_SINGLENODE()
  • Gp_role != GP_ROLE_UTILITY && Gp_role != GP_ROLE_SINGLENODE -> Gp_role != GP_ROLE_UTILITY
  • Gp_role == GP_ROLE_SINGLENODE -> IS_SINGLENODE()
  • and, finally, Gp_role == GP_ROLE_UTILITY -> Gp_role == GP_ROLE_UTILITY && !IS_SINGLENODE()
• the result is that singlenode deployment now running under GP_ROLE_UTILITY and all code already written for that code can now been leveraged. This approach has been discussed with and approved by reviewers for its simplicity and possibly smallest code change. For alternatives, see https://github.com/orgs/cloudberrydb/discussions/188#discussioncomment-6976028.

Scripts

• we plan to use the already baked NUM_PRIMARY_MIRROR_PAIRS=0 to support create singlenode deployment in all scripts and commands (gpstop, gpstart, etc.)
• all scripts (and of course, the user) can reliably detect whether the remote is in singlenode or not by gp_internal_is_singlenode. So user don't need add extra argument to the commands.

Tests

The most challenging and irksome part of this PR is how to handle tests. Many tests in regress relies on test plan which surely is different under singlenode mode and isolation2 tests often require concurrent semantic and utility mode (-1U, 1U, etc.) that singlenode does't support. This first implementation just copies and modifies all tests from regress and isolation2 into singlenode_regress and singlenode_isolation2. It's, undoubtedly, far from perfect.

FAQ

• Q: How is singlenode differs from the already-supported utility mode?
  A: To use utility mode one still needs to create a multi-segment cluster, and then connects to one of them using -c gp_role=utility. This is different from singlenode because the latter requires no segment at all from the ground up. Regarding code changes, as described above, they're mostly identical. There're a few cases where singlenode mode behaves different from utility mode and thus need special care, but they're generally rare.

Rollout/Adoption Plan

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[tuhaihe]

Hi @Ray-Eldath, thanks for submitting your proposal. You can access Proposal No.4 through the following URL: https://github.com/cloudberrydb/community/blob/main/proposals/cp-4/cp-4.md.

[Ray-Eldath]

I've experimented with both design:

use unmodified scripts, when a coordinatorOnly instance gets built, it's by default gp_role = utility

I've tested this approach last week, and it seems by far works well. Though many tests needs to be modified, we do not need to introduce significant changes to code because we can leverage all if (Gp_role == GP_ROLE_UTILITY) that're already out there. we can add special condinationals for those functionalities that're disabled in utility but should be enabled in singlenode (e.g., resgroup, auth).

go for gp_role = dispatch

this approach faces several problems, but I've came through them.

• Three global code replace need to be carried out:
  • Gp_role == GP_ROLE_UTILITY -> IS_UTILITY_OR_SINGLENODE(Gp_role), and
  • Gp_role == GP_ROLE_DISPATCH -> (Gp_role == GP_ROLE_DISPATCH && !IS_SINGLENODE()), and
  • Gp_role != GP_ROLE_DISPATCH -> (Gp_role != GP_ROLE_DISPATCH || IS_SINGLENODE())
• as you see, this approach requires the code to "act as utility mode" while it's actually in dispatch mode. this is by no means ideal.
• another problem is that pg_ctl actually relies on the gp_role=dispatch to decide whether to wait for ready signal or dtmready signal. in singlenode, it's in dispatch mode but w/o dtm, so pg_ctl will wait forever. to solve this, pg_ctl needs to read the postgres.conf. that requires a lot of new code.

---

Both design at least "works" to a limited extent. It's time to decide. So, which path should we go after? @avamingli

[avamingli]

It seems more easy for approach 1 as your spike.
And remember that zero-segments are not utility mode, it's not easy to enum all places utility can't do(auth is only an example).
If we choose approach 1,
it seems reasonable to keep a internal 'single node' role, but don't expose it to users to avoid such crashes in pr's comments.

[Ray-Eldath]

@avamingli Idk. even if we insist on getting Gp_role to be GP_ROLE_DISPATCH, the only way to make CBDB at least passes those modified tests (make installcheck-singlenode) is to make these three global code replacement, which had already disabled those functionalities that should be enabled in singlenode mode. we may still need to find out which one need to be enabled and fix them manually.

as for the internal singlenode state, I havn't tried yet, but why it's necessary...? If we go for approach 1, the Gp_role is utility, and everything just works as usual. plus, even if we add a new role, it would still acts in the same way as utility (thus disabled all those functionalities) if we just do a global replacement from Gp_role == UTILITY to IS_UTILITY_OR_SINGLENODE().

[Ray-Eldath]

After discussed with reviewers we've settled with one particular design and the propsal has been updated to reflect the newest changes.

[wnch]

I want to know how to support expanding SingleNode Deployment to multi-segment Deployment ?

[avamingli]

Hi, @wnch

Currently not supported.

[wnch]

Is it in road-map?

[avamingli]

Is it in road-map?

Not in 2024, at least.

Scale a single instance to multi sounds much like scale any segments feature which is cloud native and elastic.
We have that in our commercial version.