Watches in multiple pods and leader election

[EdwardCooke]

Is it possible to enable leader election for certain logic of my code while having watches controllers continue working when not a leader?

Initial look shows that if I enable leader election then only the leader will execute events. But. I need all pods to execute watch events to keep their local cache's in sync and have the underlying logic only execute when they are a leader.

[kimpenhaus]

hey @EdwardCooke - thanks for asking. I am not pretty sure if this is actually possible, even though there is the possibility to configure custom leader election handling by setting the LeaderElectionType to Custom:

.AddKubernetesOperator(settings =>
            {
                settings.LeaderElectionType = LeaderElectionType.Custom;
            })

This lets you implement custom leader election logic, e.g. like we do: we have to manage several namespaces and implemented a leader election based on the namespace being responsible for.

But back to your problem: in my opinion this highly depends on your implementation details and who is responsible for syncing the cache and doing the reconcile.

our implementation looks like:

public sealed class NamespacedLeaderElectionResourceWatcher<TEntity>(
    ActivitySource activitySource,
    ILogger<NamespacedLeaderElectionResourceWatcher<TEntity>> logger,
    IFusionCacheProvider cacheProvider,
    ITimedEntityQueue<TEntity> entityQueue,
    OperatorSettings settings,
    IEntityLabelSelector<TEntity> labelSelector,
    IKubernetesClient client,
    IHostApplicationLifetime hostApplicationLifetime,
    INamespaceLeadershipManager namespaceLeadershipManager,
    LeaderElector elector)
    : LeaderAwareResourceWatcher<TEntity>(
        activitySource,
        logger,
        cacheProvider,
        entityQueue,
        settings,
        labelSelector,
        client,
        hostApplicationLifetime,
        elector)
    where TEntity : IKubernetesObject<V1ObjectMeta>
{
    protected override async Task OnEventAsync(WatchEventType eventType, TEntity entity, CancellationToken cancellationToken)
    {
        if (!await namespaceLeadershipManager.IsResponsibleForNamespace(entity.Namespace(), cancellationToken))
        {
            return;
        }

        using var scope = logger.BeginScope(logger.CreateScopeForEntityLeadership(entity, namespaceLeadershipManager));
        await base.OnEventAsync(eventType, entity, cancellationToken);
    }
}

maybe you can use something similar to either call the cache sync when not being responsible (and then return) or do the base call when being responsbile.

if that helps and you need more information an what to register and overload. please let me know.