Anti-Affinity Placement Strategy for Oxia Cluster

[mattisonchao]

Anti-Affinity Placement Strategy for Oxia Cluster

Backgound

In the current implementation, we can only distribute shards to different oxia servers (nodes), which can not satisfy multi-AZ requirements.

Goals

This PR supports a label-based anti-affinity shards placement policy. Which can set the constraints for shard distribution by customized labels.

In the diagram above, we introduced three shards as examples. Two of them have an anti-affinity policy, and one of them does not.

the shard placement is as follows:

• shard-1: [node1,node3,node5]
• shard-2: [node2,node4,node6]
• shard-3: [node1,node2,node3]

Implementation

Metadata

This proposal defines a metadata and labels and policies structure to carry more information for server.

type ClusterConfig struct {
	Namespaces     []NamespaceConfig         `json:"namespaces" yaml:"namespaces"`
 	Servers        []Server                  `json:"servers" yaml:"servers"`
+	ServerMetadata map[string]ServerMetadata `json:"serverMetadata" yaml:"serverMetadata"`
}

type NamespaceConfig struct {
	Name                 string                       `json:"name" yaml:"name"`
	InitialShardCount    uint32                       `json:"initialShardCount" yaml:"initialShardCount"`
	ReplicationFactor    uint32                       `json:"replicationFactor" yaml:"replicationFactor"`
	NotificationsEnabled common.OptBooleanDefaultTrue `json:"notificationsEnabled" yaml:"notificationsEnabled"`
+	Policies             *policies.Policies           `json:"policies,omitempty" yaml:"policies,omitempty"`
}

+ type Policies struct {
+ 	AntiAffinities []AntiAffinity `json:"antiAffinities,omitempty" yaml:"antiAffinities,omitempty"`
+ }

Selector

This proposal defines a concept that can select an ensemble by different policies and implementations.

var (
	ErrUnsatisfiedAntiAffinities      = errors.New("unsatisfied anti-affinities")
	ErrUnsupportedUnsatisfiableAction = errors.New("unsupported unsatisfiable action")
)

type Selector interface {
	SelectNew(
		candidates []model.Server,
		candidatesMetadata map[string]model.ServerMetadata,
		policies *p.Policies,
		status *model.ClusterStatus,
		replicas uint32) ([]model.Server, error)
}

Merged Selector

This is the wrapper selector. It will go through the whole selector chain to choose satisfied candidates. And will ensure the selected replica quantity is expected.

Anti-Affinity Selector

The Anti-Affinity Selector uses policy-defined labels to identify and select suitable candidates for scheduling.

// UnsatisfiableAction defines the behavior when policy constraints cannot be satisfied.
type UnsatisfiableAction string

const (
	// DoNotSchedule instructs the coordinator to not schedule the shard when constraints
	// cannot be satisfied. This is the more conservative approach that ensures policy
	// compliance at the cost of potentially leaving shards unscheduled.
	DoNotSchedule UnsatisfiableAction = "DoNotSchedule"

	// ScheduleAnyway instructs the coordinator to proceed with scheduling the shard even
	// when constraints cannot be satisfied. This is a more permissive approach that
	// ensures service availability at the cost of potentially violating policy constraints.
	ScheduleAnyway UnsatisfiableAction = "ScheduleAnyway"
)

// AntiAffinity defines rules for keeping certain shards separated based on labels.
// It helps in achieving better fault tolerance by ensuring shards are distributed
// across different failure domains.
type AntiAffinity struct {

	// Labels specifies the set of labels to consider when making anti-affinity
	// decisions. Shards with matching labels will be scheduled according to
	// the anti-affinity rules.
	Labels []string `json:"labels,omitempty" yaml:"labels,omitempty"`

	// UnsatisfiableAction determines what action to take when anti-affinity
	// constraints cannot be satisfied. It can be either DoNotSchedule or
	// ScheduleAnyway.
	UnsatisfiableAction UnsatisfiableAction `json:"UnsatisfiableAction,omitempty" yaml:"unsatisfiableAction,omitempty"`
}

ServerIdx Selector

In scenarios where the number of satisfied candidates exceeds the expected number, the ServerIdxSelector utilises incremental seeds to select the desired quantity of ensembles.

Observeralbility

• Copy all the server metadata labels to oxia_coordinator_node_running metric.
• Support shard ensembles ID labels for leaderElectionLatency and leaderElectionsFailed metrics.

Backward & Forward Compatibility

No anti-affinity policy

There's no side effect if the existing cluster didn't configure the anti-affinity policy.

Configured anti-affinity policy

The new policy only applies to load balancing when moving shards to other nodes. Therefore, it does not affect leader election, but only the shard moving itself for the existing shards.