BFD status never invalidated when no chassis can run the session — ECMP keeps routing to a dead nexthop

[remeq]

Summary

When the chassis running a BFD session (created for BFD-monitored ECMP
static routes) goes away and no other chassis has the bridge mapping needed
to take the gateway port over, the session's status in both the
Northbound BFD table and the Southbound bfd table keeps its last
reported value — typically up — indefinitely. Logical router ECMP route
selection trusts that stale status, so traffic hashed to the dead nexthop
is blackholed with no automatic recovery for as long as the outage lasts.

The status also never transitions during the outage, so the event cannot
even be detected by monitoring the NB/SB BFD tables.

Versions

Reproduced on 24.03.6 and on branch-24.03 tip (dd45c93); the relevant code
is unchanged on current main. Line references below are for branch-24.03
@ dd45c93.

Reproduction

Topology: 2 chassis with asymmetric bridge mappings — chassis A
ovn-bridge-mappings=physnet1:br-1, chassis B physnet2:br-2 (each
physnet exists on exactly one chassis; both enable-chassis-as-gw).
A logical router with two distributed gateway ports (one per physnet, each
pinned to its chassis via HA_Chassis_Group), two ECMP default routes with
bfd set, and an external BFD peer (FRR) on each physnet.

1. Verify both sessions are up on both ends and ECMP spreads flows
   across both nexthops.
2. Stop chassis B entirely (systemctl stop ovn-controller ovs-vswitchd ovsdb-server). Its Chassis row is removed on graceful exit and the
   gateway port's chassisredirect Port_Binding ends up with chassis: []
   — no chassis owns the mapping.
3. The FRR peer detects the failure within its detect time
   (Status: down, "control detection time expired").
4. On the OVN side, indefinitely (verified for minutes; production
   sightings of 30+ minutes — it is unbounded):

$ ovn-nbctl --columns=dst_ip,status list BFD
dst_ip              : "192.168.1.201"
status              : up
dst_ip              : "192.168.2.201"
status              : up        <-- peer sees down; nobody runs this session

SB.bfd shows the same stale up.
5. Data plane: of 8 test destinations from a VM behind the router, the
ones hashing to the dead nexthop time out, the others keep working —
ECMP still selects the dead gateway. No automatic recovery while the
mapping is absent.

Root cause

• The only writer of SB.bfd.status is the chassis-local ovn-controller:
  bfd_monitor_run() (controller/pinctrl.c:7810) writes on local BFD
  state-machine transitions. A chassis that is gone writes nothing.
• northd never derives status; build_bfd_table() (northd/northd.c:10184)
  only mirrors SB→NB (nbrec_bfd_set_status(nb_bt, bfd_e->sb_bt->status),
  northd.c:10245).
• The guard if (op && op->sb && op->sb->chassis) (northd.c:10228) simply
  skips ports with no chassis; there is no else-branch. The only paths
  writing "down" convert an existing admin_down.

So no code path exists that could ever mark the session down in this
situation. (Additionally, that guard checks the LRP's own Port_Binding,
which for distributed gateway ports is type patch and never has a
chassis — which is also why SB.bfd.chassis_name stays empty even for
healthy sessions on the main use case.)

The session-ownership rule lives in bfd_monitor_run(): a chassis runs the
session iff the port's chassisredirect twin (cr-<port>) is resident on
it, or the port is an l3gateway port bound to it. northd has all the
information needed to evaluate that rule from the databases and conclude
"nobody can be running this session."

Expected behavior

When no chassis can run a BFD session, its status should not stay up.
Marking it down is sufficient for correct routing behavior: route
parsing already drops BFD-monitored routes whose session is down
(parsed_routes_add(), northd.c:10715-10735), so the ECMP nexthop is
withdrawn with no routing-code changes. Recovery composes with the
existing controller logic: when a chassis (re)claims the port,
ovn-controller starts a fresh state machine and reports up on
negotiation as today.

---

Analysis and patch developed with assistance from Claude Code.

[remeq]

The hard-crash/liveness follow-up described in this issue is now published as the Concept C stack:

• northd: Mark BFD sessions down when no chassis can run them. #321 derives BFD down when no chassis can run the session.
• controller: Reconcile BFD status with local session state. #323 reconciles a live controller local BFD state with a stale SB status.
• northd: Treat HA group membership as BFD eligibility #325 makes stale HA-group membership ineligible for BFD ownership (draft, stacked on the first two).
• Neutron RFE: https://bugs.launchpad.net/bugs/2165165
• Neutron implementation: https://review.opendev.org/c/openstack/neutron/+/1002536

The Neutron half is opt-in and fail-closed. It observes fresh Chassis_Private.nb_cfg generation progress with a local monotonic hold-down, changes only BFD-only router HCGs, persists exact exclusions/priorities, skips mixed/LSP-referenced groups, and refuses a global drain. Together, the CMS decides liveness eligibility and northd derives BFD truth without adding a new BFD status writer or schema field.