Scheduler logs wrong replica-role after failover

[gabesaba]

What you expected to happen:
This value is only set once, upon Start:

kueue/pkg/scheduler/scheduler.go

Lines 165 to 171 in 3b299a3

	// Start implements the Runnable interface to run scheduler as a controller.
	func (s *Scheduler) Start(ctx context.Context) error {
	log := roletracker.WithReplicaRole(ctrl.LoggerFrom(ctx).WithName("scheduler"), s.roleTracker)
	ctx = ctrl.LoggerInto(ctx, log)
	go wait.UntilWithBackoff(ctx, s.schedule)
	return nil
	}

If the leader fails, and a follower becomes new leader, it has the wrong log

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Kueue version (use git describe --tags --dirty --always):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[gabesaba]

This Start method should only be called once the replica is the leader. However, I was observing logs where replica-role=follower for scheduler.

[gabesaba]

Perhaps a race, where failover happens, but the role tracker doesn't reflect the new value until after the Start method is called

[mimowo]

Good point, it might be related to other issues we've seen: #9287 and #9288

cc @IrvingMg