Kminion goes into crashback loop on Confluent Cloud Fully Managed

[reidmeyer]

Kminion puts a partition on each broker so that it can accurately measure latency on each broker.
If Kminion detects that a rebalancing of the cluster has caused there to be a situation other than 1 partition on each broker, Kminion will attempt to rearrange the partitions. On Confluent Cloud Fully Managed, Kminion can not get rights to do this action. Kminion then goes into a crash back loop. To fix, I delete the topic, and let Kminion recreate it.

I was thinking, it could be a feature for Kminion to delete/recreate the topic given this scenario, rather than requiring manual intervention.

[reidmeyer]

Or maybe I'm mistaken and I just need a certain ACL I'm missing.

I'm guessing ALTER on CLUSTER is what's required?

Pretty sure, even with this ACL, confluent blocks reassignment calls.

[weeco]

Hey @reidmeyer ,
yes seems possible that managed providers block reassign partitions because they possibly have their own preference how to place replicas on the brokers. Cluster alter sounds like the right permission to me. I think adding an option to re-create the topic to re-balance would be a bit of a hack in kminion because you'd also need to recreate the consumer group, re-init producers etc., it's kind of a full kminion restart

[reidmeyer]

Definitely also feels like a bit of a hack.
As of now, deleting the end-to-end topic, and letting kminion recreate it has been our quick fix when the cluster is rebalanced.
I wonder if that works because creating topics with replica assignments is allowed, or if I'm just lucky and the partitions are always evenly distributed.

Do you have any other thoughts how kminion end-to-end functionality could work in a system that doesn't allow partition reassignment?

I'm of course also talking with Confluent about this.

[weeco]

If you don't have the control how the replicas (and leaders) are placed on the cluster, you cannot ensure that you are testing all brokers effectively and you are dependent on the luck how they were placed. Ideally managed providers allow you to disable auto-rebalancing on a per-topic basis for this specific usecase, but they may argue that SLA measuring is their responsibility as part of the offering and I don't see other usecases for disabling (smart) auto rebalancing.

[reidmeyer]

Can you confirm, when kminion creates the end-to-end topic, does it decide where those partitions go, or does it let kafka decide, and then repartition if necessary?

[weeco]

@reidmeyer I just looked it up , currently it lets Kafka decide where to place the replicas but right after creation it checks whether the replicas should be reassigned.

However, this changes with this PR: #312 (significantly rewrite of the replica assignment logic). You can give this docker image a try if you think it will solve your problems. Relevant code piece:

kminion/e2e/topic.go

Lines 292 to 312 in 25b479c

	// Create topic with specific replica assignments from the planner
	topic := kmsg.NewCreateTopicsRequestTopic()
	topic.Topic = topicCfg.Name
	topic.NumPartitions = -1 // Must be -1 when using ReplicaAssignment
	topic.ReplicationFactor = -1 // Must be -1 when using ReplicaAssignment
	topic.Configs = createTopicConfig(topicCfg)
	// Convert planner's CreateAssignments to Kafka's ReplicaAssignment format
	for i, assignment := range plan.CreateAssignments {
	replica := kmsg.NewCreateTopicsRequestTopicReplicaAssignment()
	replica.Partition = int32(i)
	replica.Replicas = append([]int32(nil), assignment.Replicas...)
	topic.ReplicaAssignment = append(topic.ReplicaAssignment, replica)
	}
	req := kmsg.NewCreateTopicsRequest()
	req.Topics = []kmsg.CreateTopicsRequestTopic{topic}
	res, err := req.RequestWith(ctx, s.client)
	if err != nil {
	return fmt.Errorf("failed to create e2e topic: %w", err)

[reidmeyer]

I'm so impressed with your quick response.
So in the new implementation of that PR, it decides the partitions on topic creation, using replica assignment?

This is definitely attractive, but my guess is this won't solve my problem, because doing a quick test creating a topic on confluent cloud with replica-assignment, it looks like it wasn't respected. My guess is that confluent ignores the replica assignment request. But this has been very helpful in understanding the situation.

[weeco]

So in the new implementation of that PR, it decides the partitions on topic creation, using replica assignment?

To be precise it decides on the replica assignments (a map where all replicas of each partition should be placed on the brokers)

[reidmeyer]

Thanks
I've just confirmed that sadly won't solve my problem since Confluent ignores replica assignment commands.
Nonetheless, I think this is obviously not really a kminion problem.
Also the feature I requested also doesn't make sense in that recreating the topic doesn't necessarily fix the partition assignment.

The one question I still have: I've seen, which the cluster has a partition reassignment take place, I've seen kminion go into a crashbackloop if it's unable to reassign the partitions. Is this the desired outcome? Perhaps it's also reasonable to keep kminion running, even if it can't balance the partitions across the brokers. I've seen It sometimes where it does a crashback loop, and sometimes where it doesn't (I guess it doesn't crash if the topic is created at startup by kminion, even if the replica assignment isn't as desired)