Improving Device Discovery and Filtering Control in DRANET

[gauravkghildiyal]

DRANET currently uses a static --filter flag (CEL expression) to decide which network devices to publish. While this works for simple cases, it is proving too rigid for complex cloud environments.

The current "static" model has two main limitations:

1. No Per-Node Flexibility: The filter is set in the manifest and is the same for every node in a pool, making it impossible to handle node-specific hardware state.
2. Attribute Bloat: To make the filter "dynamic," we currently have to add extra attributes to devices just so the filter can see them. This can lead to unnecessary bloat in the ResourceSlice, and we are already approaching the limit on the max number of attributes allowed per device.

Why we need it

Recent discussions (e.g. #138 (comment), #138 (comment)) have shown that standard logic (like "exclude anything with a default gateway") isn't enough. On these platforms, multiple interfaces might carry default routes even if they are meant for workload traffic.

Possible approaches (amongst others, feel free to comment)

Instead of just letting cloud providers add attributes to devices, we could allow them to filter the device list during discovery.

This keeps the logic simple and localized:

• Cloud providers can use their existing knowledge (metadata, specific hardware IDs) to explicitly exclude management or host-reserved NICs.
• We avoid "special-casing" new interfaces for every unique cloud side-effect.
• We keep the ResourceSlice clean by not publishing "internal-only" attributes just for filtering purposes.

/cc @aojea @dkennetzoracle @tamilmani1989

[aojea]

Ah, so you mean the cloud provider has a way to return the dynamic filter cel expression?

[dkennetzoracle]

To me this reads a bit backwards:

Cloud providers can use their existing knowledge (metadata, specific hardware IDs) to explicitly exclude management or host-reserved NICs.

OKE needs the opposite of exclusion, while I think Azure does need exclusion which I think is addressed by @aojea PR corrections. But generally speaking, a fairly robust fix that could providers could extend and could participate in device discovery could be a 3 state filter like:

  // ProviderDeviceFilter lets a CloudInstance participate in discovery filtering.
  // Returning nil means "no opinion" — dranet's default rules apply. Non-nil
  // values force inclusion (true) or exclusion (false).
  type ProviderDeviceFilter interface {
      FilterDevice(id DeviceIdentifiers) *bool
  }

This could end up being the one interface that rules them all for CSPs such that any custom filtering inclusion / exclusion logic could be attached... :)

the nil case would mean "the provider has no opinion on a discovered device, fall back to dranet logic". Covers

• Loopback, docker0, bridges, VLANs

I could see concretely based on my and @tamilmani1989 's cases having simple interfaces which would exist during discovery:

OKE:

func (o *OKEInstance) FilterDevice(id DeviceIdentifiers) *bool {                                                                                                                                                                                                             
    if o.GpuMemoryFabric != "" && id.RDMA {                                                                                                                                                                                                                                  
        t := true
        return &t  // force include despite RA default route                                                                                                                                                                                                                 
    }                                                                                                                                                                                                                                                                        
    return nil     // everything else: defer to dranet's default heuristic                                                                                                                                                                                                   
}

Azure:

func (a *AzureInstance) FilterDevice(id DeviceIdentifiers) *bool {
    if id.MAC == a.primaryNICMAC {                                                                                                                                                                                                                                           
        f := false
        return &f  // force exclude — we know this is the VM's primary                                                                                                                                                                                                       
    }                                                                                                                                                                                                                                                                        
    return nil     // including the secondary NIC: let dranet's heuristic decide
}

[tamilmani1989]

I liked what @dkennetzoracle suggested. Looks straightforward and simpler to me.

@gauravkghildiyal I assume the reason you suggested dynamic cel is that it reuses existing code and avoids introducing new abstractions. I have few concerns with that approach:

• Today -filter is cluster-scoped (set once at deployment via daemonset args) but a cloud provider-returned CEL expression would be node specific. These operate at fundamentally different levels.

• If a user has already specified a --filter and the cloud provider also returns a dynamic CEL expression, how do we reconcile the two? Do we AND them together, does one take precedence, or do we reject entirely?

• Each cloud provider would need to understand the specific device attributes and construct CEL expressions accordingly. Different providers may rely on different attributes. If an attribute is deprecated, renamed, or its format changes, every CEL expression referencing must be updated. This creates a migration and backward compatibility concern that would be easier to manage with a more structured API. wdyt?

[gauravkghildiyal]

@tamilmani1989 -- I agree, to be clear, I'm not in favor of introducing another CEL-based mechanism here. CEL is excellent for static configuration in manifests or flags, but we clearly are not bounded by that here and hence should definitely use a more structured Go API based approach.

What I'm thinking of is something a bit more powerful than just filtering. As we move towards making use of newer DRA API constructs like DRA Device Taints (KEP-5055), the responsibilities of cloud providers during discovery will likely expand. They will need to perform multiple actions, including:

1. Add Attributes (already exists)
2. Filter Devices (being discussed here)
3. Taint Devices (upcoming)

So instead of independent hooks for each task, perhaps a single generic modifier like ProcessDevices could be used that creates a stable extension point and allows cloud providers to act on the device list in a single pass.

[dkennetzoracle]

It's an interesting proposal, and I like the idea at the broader (outer) scope for DRANET. You have an API that is ProcessDevices, but I think the 3 below are distinct pieces of logic. Interfaces as contracts are a bit restrictive but could be nice here if we want to ensure CSPs maintain "compliance" to the logic.

  type ProcessProviderDevice interface {
      AddAttributes(id DeviceIdentifiers) *bool
      FilterDevices(id DeviceIdentifiers) *bool
      TaintDevices(id DeviceIdentifiers) *bool
  }