Description
On a VTEP carrying multiple L3VNIs (symmetric IRB, one VRF per tenant), if the L3VNI SVIs (bridge devices) are configured with different MAC addresses per VRF, cross-VTEP routed traffic for at least one of the VRFs is silently blackholed on the receiving side. No warning or log is emitted; type-5 routes are present and look healthy in show bgp l2vpn evpn and show ip route vrf ... — packets just disappear.
Giving all L3VNI SVIs on the VTEP a single shared system MAC resolves it completely (matching the Cumulus "single system MAC per VTEP" guidance). Our suspicion is last-write-wins handling of the remote router-MAC per VTEP in zebra when the advertised RMACs differ per VNI from one VTEP-IP — but we haven't confirmed the mechanism; the ask may reduce to "warn loudly when one VTEP advertises conflicting RMACs".
Steps to reproduce
Two containers (VTEP A, VTEP B) + one route server/leaf, all FRR 10.2.1:
- On each VTEP: two VRFs (
vrf-blue vni 100100, vrf-green vni 100200), each with br-vrf-X (SVI) + vniX VXLAN device, advertise-all-vni, per-VRF router bgp ... vrf instances with advertise ipv4 unicast under l2vpn evpn.
- Assign different MACs to
br-vrf-blue and br-vrf-green on VTEP A (e.g. 02:de:01:62:00:01 / 02:de:01:62:00:02). Keep an interface with a connected route in each VRF.
- From an address in vrf-green on VTEP B, ping the vrf-green address on VTEP A (after arranging route leak or intra-VRF reachability).
Expected: replies (routes are installed and valid on both sides).
Actual: 100% loss for one of the VRFs; show ip route vrf shows the route as best/installed; nothing logged.
With step 2 changed to one shared MAC for both SVIs: 0% loss.
We hit this in a NICo/HBN-style lab (the design BlueField HBN uses) and can share a docker-compose reproducer (FRR 10.2.1 official images) if helpful.
Version
FRR 10.2.1 (quay.io/frrouting/frr:10.2.1), Linux 6.x (Ubuntu 24.04 container host), VXLAN/EVPN with head-end replication, eBGP unnumbered underlay.
Ask
Either handle per-VNI RMACs from one VTEP correctly, or detect the conflict and log a clear warning — the silent blackhole with healthy-looking control plane is the painful part.
Description
On a VTEP carrying multiple L3VNIs (symmetric IRB, one VRF per tenant), if the L3VNI SVIs (bridge devices) are configured with different MAC addresses per VRF, cross-VTEP routed traffic for at least one of the VRFs is silently blackholed on the receiving side. No warning or log is emitted; type-5 routes are present and look healthy in
show bgp l2vpn evpnandshow ip route vrf ...— packets just disappear.Giving all L3VNI SVIs on the VTEP a single shared system MAC resolves it completely (matching the Cumulus "single system MAC per VTEP" guidance). Our suspicion is last-write-wins handling of the remote router-MAC per VTEP in zebra when the advertised RMACs differ per VNI from one VTEP-IP — but we haven't confirmed the mechanism; the ask may reduce to "warn loudly when one VTEP advertises conflicting RMACs".
Steps to reproduce
Two containers (VTEP A, VTEP B) + one route server/leaf, all FRR 10.2.1:
vrf-bluevni 100100,vrf-greenvni 100200), each withbr-vrf-X(SVI) +vniXVXLAN device,advertise-all-vni, per-VRFrouter bgp ... vrfinstances withadvertise ipv4 unicastunder l2vpn evpn.br-vrf-blueandbr-vrf-greenon VTEP A (e.g. 02:de:01:62:00:01 / 02:de:01:62:00:02). Keep an interface with a connected route in each VRF.Expected: replies (routes are installed and valid on both sides).
Actual: 100% loss for one of the VRFs;
show ip route vrfshows the route as best/installed; nothing logged.With step 2 changed to one shared MAC for both SVIs: 0% loss.
We hit this in a NICo/HBN-style lab (the design BlueField HBN uses) and can share a docker-compose reproducer (FRR 10.2.1 official images) if helpful.
Version
FRR 10.2.1 (quay.io/frrouting/frr:10.2.1), Linux 6.x (Ubuntu 24.04 container host), VXLAN/EVPN with head-end replication, eBGP unnumbered underlay.
Ask
Either handle per-VNI RMACs from one VTEP correctly, or detect the conflict and log a clear warning — the silent blackhole with healthy-looking control plane is the painful part.