northd: Drop post-NF flood copies causing MAC flaps.

A Network Function (NF) in inline mode redirects matched traffic
through a service VM.  When the redirected traffic is IP
unknown-unicast (destination MAC not yet in the FDB), the packet
coming back from the NF is re-flooded by the switch, because the
destination MAC is still unknown.  That re-flood produces a copy on
the same port the packet originally arrived on, causing MAC flaps and
potential L2 loops.

Following are the example packet flows.

Example 1, VLAN switch (MAC flap):

  Topology
    VM1, VM2 and the NF are on the same logical switch (LS), which is
    VLAN-backed (localnet port on every node).  VM1 is on N1, VM2 on
    N2 (its port has "unknown" in addresses and a to-lport ACL that
    redirects to the inline NF on N3).

  Flow
    1. VM1 sends pkt to dst MAC X (not in FDB).
    2. The LS floods the pkt; on N1 the copy exits the localnet port
       and the TOR floods it to N2.
    3. On N2 the pkt ingresses on localnet; the LS floods it and the
       copy reaches VM2 (unknown-addr).
    4. The ACL redirects the pkt to the NF on N3; the NF returns it
       to N2.
    5. X is still not in the FDB, so the LS floods again; on N2 one
       copy exits the localnet port.

  Result
    The TOR now sees VM1's source MAC on N2's port, but it had just
    learned VM1's MAC on N1's port, so the MAC flaps on the TOR.

Example 2, VLAN switch with two protected VMs (loop):

  Topology
    As above, plus VM3 (on N3, same LS) also has "unknown" in
    addresses and is also NF-protected.

  Flow
    1-5 as above; in parallel, the original flood also reaches N3 via
        the TOR, where it is redirected to N3's NF, returns, and the
        LS re-floods it out N3's localnet port.
    6. N3's re-flood reaches N2 via the TOR; the LS floods it on N2,
       the copy hits VM2 (unknown-addr), is redirected to the NF, and
       re-floods out N2's localnet port.
    7. That re-flood reaches N3 via the TOR; the LS floods it on N3,
       the copy hits VM3 (also unknown-addr), is redirected to the
       NF, re-floods out localnet, ...

  Result
    The pkt keeps bouncing between N2 and N3 via the TOR, i.e. an
    L2 loop, on top of continuous MAC flaps.  The loop persists as
    long as X stays unknown.

Example 3, overlay switch (copy reflected to inport):

  Topology
    VM1, VM2 and the NF are on the same LS and all on N1.  VM1 and
    VM2 both have "unknown" in addresses; VM2's to-lport ACL
    redirects to the inline NF.

  Flow
    1. VM1 sends pkt to dst MAC X (not in FDB).
    2. The LS floods to MC_UNKNOWN members on N1; one copy goes to
       VM2.
    3. VM2's ACL redirects its copy to the NF; the NF returns it to
       N1.
    4. X is still not in the FDB, so the LS floods to MC_UNKNOWN
       again; one copy is headed back out VM1's own port.

  Result
    VM1 receives a copy of the packet it just sent (reflected to the
    inport).

Fix:

Use the nf_learn_orig_inport() / nf_lookup_orig_inport() actions
from the previous commit to remember the original ingress port and
drop the copy if it is about to be sent back out of that port.

  - In the NF redirect stage (ls_in_nf for a from-lport ACL,
    ls_out_nf for a to-lport ACL), existing priority-99 redirect
    flows are updated to learn the original ingress port
    (nf_learn_orig_inport()) before redirecting unicast IP packets
    to the NF.

  - Priority 50 (overlay switches only, in ls_in_nf): learn on
    the inport's ingress node when the ACL did not redirect
    the packet (REGBIT_NF_ENABLED == 0), so the learn stays
    co-located with the lookup even when the redirecting port and
    the inport live on different nodes.  VLAN-backed switches
    do not need this: the post-NF packet is always sent back to
    the inport's node.

  - On the post-NF return path, run
    REGBIT_NF_LOOKUP_HIT = nf_lookup_orig_inport() on both NF ports.
    A priority-115 flow in ls_out_pre_acl runs the lookup on
    input_port and skips already-traversed egress stages.  A
    priority-2 flow in ls_out_nf runs the lookup on output_port for
    packets redirected from the ingress pipeline (for example, a
    from-lport request that is re-flooded after NF processing and
    re-enters egress).

  - A new priority-110 flow in ls_out_check_port_sec drops packets
    with REGBIT_NF_LOOKUP_HIT == 1, i.e. the copies about to be
    sent back out of the port they originally arrived on.

All new flows are gated on the switch having an inline NF group on
an ACL.

Acked-by: Aditya Mehakare <aditya.mehakare@nutanix.com>
Fixes: 8e2d6fa ("northd, tests: Network Function insertion logical flow programming.")
CC: Sragdhara Datta Chaudhuri <sragdha.chaudhu@nutanix.com>
Assisted-by: Claude Opus 4.7, Cursor
Signed-off-by: Naveen Yerramneni <naveen.yerramneni@nutanix.com>
Signed-off-by: Dumitru Ceara <dceara@redhat.com>

Author: naveen-yerramneni

Documentation/ref/ovn-logical-flows.7.rst

@@ -946,11 +946,15 @@ refer to either the parent or child ports as applicable to this logical switch.
   table.
 
 - In inline: For each active network function with *id* that is referenced in a
-  network function group, a priority-99 flow matches ``reg8[21] == 1 &&
-  reg8[22] == 1 && reg0[22..29] == id`` and sets ``outport=P; output;`` where
-  *P* is the ``inport`` of that network function. This redirects request
-  packets for flows matching ``from-lport`` ACLs with network_function_group
-  to the specific network function selected by the Pre Network Function stage.
+  network function group, priority-99 flows match ``ip4 && reg8[21] == 1 &&
+  reg8[22] == 1 && reg0[22..29] == id`` and ``ip6 && reg8[21] == 1 &&
+  reg8[22] == 1 && reg0[22..29] == id`` and set
+  ``nf_learn_orig_inport(ipv6 = false|true); outport=P; output;``
+  where *P* is the ``inport`` of that network function. This redirects request
+  packets for flows matching ``from-lport`` ACLs with
+  network_function_group to the specific network function selected by the Pre
+  Network Function stage. It also learns the original inport before
+  redirection.
 
 - In vtap mode: For each active network function with *id*, a priority-99
   forward flow matches ``reg8[21] == 1 && reg8[22] == 1 && reg0[22..29] == id``
@@ -965,6 +969,12 @@ refer to either the parent or child ports as applicable to this logical switch.
   This redirects response and related packets for ``to-lport`` ACLs to the
   same network function that handled the request.
 
+- In inline (overlay logical switches only), two priority-50 flows match
+  ``ip4 && reg8[21] == 0`` and ``ip6 && reg8[21] == 0`` and set
+  ``nf_learn_orig_inport(ipv6 = false|true); next;``. This learns on the
+  packet's ingress node even when redirection happens on a different node,
+  so the learn stays co-located with the lookup.
+
 - In vtap mode: A priority-99 reverse flow matches ``reg8[21] == 1 && reg8[22]
   == 0 && ct_label.nf_id == id`` and sets ``clone { outport = P; output; };
   next;`` to mirror response/related packets to the same NF.
@@ -1645,13 +1655,17 @@ peer of a logical router port. This flow is added to skip the connection
 tracking of packets which will be entering logical router datapath from logical
 switch datapath for routing.
 
-This table also has a priority-110 flow for each network_function ``inport`` *P*
-that matches ``inport == P``. The action is to skip all the egress tables up to
-the :ref:`Network Function <ls-out-13>` table and advance the packet directly to
-the table after that. This is for the case where packet redirection happens in
-egress :ref:`Network Function <ls-out-13>` table. The same packet when it comes
-out of the other port of network function, they should not be processed again by
-the same egress stages, specially they should skip the conntrack processing.
+This table also has a priority-115 flow for each network_function ``inport``
+*P*
+that matches ``inport == P`` and applies
+``reg8[24] = nf_lookup_orig_inport(); next(pipeline=egress, table=T);`` where
+*T* is the table immediately following :ref:`Network Function <ls-out-13>`.
+This is for the case where packet redirection happens in egress
+:ref:`Network Function <ls-out-13>` table. When the packet comes back from the
+other NF port, it should not be processed again by the same egress stages,
+specifically conntrack processing should be skipped. This flow also performs
+``nf_lookup_orig_inport()`` before continuing so post-NF loopback copies can
+be detected and dropped.
 
 .. _ls-out-3:
 
@@ -1886,13 +1900,15 @@ in ``ct_label.nf_id`` during request processing.
   table.
 
 - In inline: For each active network function with *id* that is referenced in a
-  network function group, a priority-99 flow matches ``reg8[21] == 1 &&
-  reg8[22] == 1 && reg0[22..29] == id`` and sets ``outport=P; reg8[23] = 1;
-  next(pipeline=ingress, table=T)`` where *P* is the ``outport`` of
-  that network function and *T* is the ingress table :ref:`Destination Lookup
-  <ls-in-33>`. This redirects request packets matching ``to-lport`` ACLs with
-  network_function_group to the specific network function selected by the Pre
-  Network Function stage.
+  network function group, priority-99 flows match ``ip4 && reg8[21] == 1 &&
+  reg8[22] == 1 && reg0[22..29] == id`` and ``ip6 && reg8[21] == 1 &&
+  reg8[22] == 1 && reg0[22..29] == id`` and set
+  ``nf_learn_orig_inport(ipv6 = false|true); outport=P; reg8[23] = 1;
+  next(pipeline=ingress, table=T);`` where *P* is the NF outport and *T* is
+  the ingress :ref:`Destination Lookup <ls-in-33>` table. This redirects
+  request packets matching ``to-lport`` ACLs with network_function_group to the
+  specific network function selected by the Pre Network Function stage, and
+  learns the original inport before redirection.
 
 - In vtap mode: For each active network function with *id*, a priority-99
   forward flow matches ``reg8[21] == 1 && reg8[22] == 1 && reg0[22..29] == id``
@@ -1922,6 +1938,10 @@ in ``ct_label.nf_id`` during request processing.
   port) and advances to the next table so packets to the NF are not mirrored
   again.
 
+- In inline: A priority-2 flow matches ``inport == P`` where *P* is the NF
+  ``outport`` and sets ``reg8[24] = nf_lookup_orig_inport(); next;``.  This
+  captures post-NF packets that re-enter egress from ingress recirculation.
+
 - In vtap mode: In egress Pre ACL table, a priority-110 flow matches ``ip &&
   outport == P`` with action ``ct_clear; next;`` for the vtap NF port so
   packets toward the NF are not committed to conntrack.
@@ -1944,6 +1964,9 @@ This is similar to the port security logic in table :ref:`Ingress Port Security
 check <ls-in-0>` except that action ``check_out_port_sec`` is used to check the
 port security rules.  This table adds the below logical flows.
 
+- A priority-110 flow matching ``reg8[24] == 1`` with action ``drop;``.  This
+  drops post-NF loopback copies detected by ``nf_lookup_orig_inport()``.
+
 - A priority 100 flow which matches on the multicast traffic and applies the
   action ``REGBIT_PORT_SEC_DROP" = 0; next;"`` to skip the out port security
   checks.

TODO.rst

@@ -177,6 +177,16 @@ OVN To-do List
   * Geneve tunnel is used for supporting this feature for VLAN network.
     Extend the support over VxLAN tunnel as well.
 
+  * Inline-NF loop prevention has limitations in topologies where
+    a post-NF flood copy may reach the same logical inport from a
+    different node than the packet's original ingress node.  In that
+    case, the lookup runs on a different node than where the inport
+    was learned, so loopback copies are not identified.
+    Known example: OVN-IC L2 stretch with the NF and IC gateway on
+    different chassis.  With ``unknown``-addressed ports, the post-NF
+    packet can be flooded across AZs and come back to the source AZ via
+    another node, so the lookup does not hit the original learn entry.
+
 * CI
 
   * ovn-kubernetes: Only a subset of the ovn-kubernetes features is currently

northd/northd.c

@@ -176,6 +176,9 @@ static bool vxlan_mode;
 #define REGBIT_NF_ENABLED         "reg8[21]"
 #define REGBIT_NF_ORIG_DIR        "reg8[22]"
 #define REGBIT_NF_EGRESS_LOOPBACK "reg8[23]"
+/* Set on a post-NF packet flowing back out its original ingress port;
+ * such packets are dropped. */
+#define REGBIT_NF_LOOKUP_HIT      "reg8[24]"
 /* Register to store the network function group id */
 #define REG_NF_GROUP_ID           "reg0[22..29]"
 /* REG_NF_ID overrides REG_NF_GROUP_ID in the pre_network_function stage. */
@@ -315,6 +318,8 @@ static const char *reg_ct_state[] = {
  * |    | REGBIT_NF_{ENABLED/ORIG_DIR/                 | G |                                   |
  * |    |            EGRESS_LOOPBACK}                  | 4 |                                   |
  * |    |       (>= ACL_EVAL* && <= NF*)               |   |                                   |
+ * |    | REGBIT_NF_LOOKUP_HIT                         |   |                                   |
+ * |    |     (>= OUT_PRE_ACL && <= OUT_CHECK_PORT_SEC)|   |                                   |
  * +----+----------------------------------------------+   +-----------------------------------+
  * | R9 |              OBS_POINT_ID_EST                |   |                                   |
  * |    |       (>= ACL_EVAL* && <= ACL_ACTION*)       |   |                                   |
@@ -19776,35 +19781,64 @@ consider_network_function_inline(struct lflow_table *lflows,
 
     /* Add forward flows for redirection:
      * Flows to handle request packets for new or existing connections.
+     * These flows learn the original inport before redirection.
      *
      * from-lport ACL in_nf priority 99:
      * in_acl_eval has already categorized it and populated nf_enabled,
      * direction and nfg_id registers. in_pre_nf sets the active network
-     * function id in nf_id register. Here this rule sets the outport to the
-     * NF port and does output action to skip the rest of the ingress pipeline.
+     * function id in nf_id register. Here this rule learns original inport,
+     * sets the outport to the NF port and does output action to skip the
+     * rest of the ingress pipeline.
      *
      * to-lport ACL out_nf priority 99:
      * out_acl_eval, and out_pre_nf set the nf related registers. Then the
-     * out_nf stage sets the outport to NF port and submits the
-     * packet back to ingress pipeline l2_lkup table. The l2_lkup would skip
-     * mac based lookup as the NF_EGRESS_LOOPBACK is set.
+     * out_nf stage learns original inport, sets the outport to NF port and
+     * submits the packet back to ingress pipeline l2_lkup table. The l2_lkup
+     * would skip mac based lookup as the NF_EGRESS_LOOPBACK is set.
      */
+    ds_put_format(&match, REGBIT_NF_ENABLED" == 1 && "
+                          REGBIT_NF_ORIG_DIR" == 1 && "
+                          REG_NF_ID " == %"PRIu8,
+                  (uint8_t) nf->id);
+    size_t fwd_match_len = match.length;
+
+    /* IPv4 redirect flow: learn original inport before redirect. */
+    ds_put_format(&match, " && ip4");
+    ds_clear(&action);
     if (ingress) {
-        ds_put_format(&action, "outport = %s; output;",
+        ds_put_format(&action, "nf_learn_orig_inport(ipv6 = false); "
+                               "outport = %s; output;",
                       redirect_port->json_key);
     } else {
-        ds_put_format(&action, "outport = %s; "
+        ds_put_format(&action, "nf_learn_orig_inport(ipv6 = false); "
+                               "outport = %s; "
+                               REGBIT_NF_EGRESS_LOOPBACK" = 1; "
+                               "next(pipeline=ingress, table=%d);",
+                      redirect_port->json_key,
+                      ovn_stage_get_table(S_SWITCH_IN_L2_LKUP));
+    }
+    ovn_lflow_add(lflows, od, fwd_stage, 99, ds_cstr(&match),
+                  ds_cstr(&action), lflow_ref);
+    ds_truncate(&match, fwd_match_len);
+
+    /* IPv6 redirect flow: learn original inport before redirect. */
+    ds_put_format(&match, " && ip6");
+    ds_clear(&action);
+    if (ingress) {
+        ds_put_format(&action, "nf_learn_orig_inport(ipv6 = true); "
+                               "outport = %s; output;",
+                      redirect_port->json_key);
+    } else {
+        ds_put_format(&action, "nf_learn_orig_inport(ipv6 = true); "
+                               "outport = %s; "
                                REGBIT_NF_EGRESS_LOOPBACK" = 1; "
                                "next(pipeline=ingress, table=%d);",
                       redirect_port->json_key,
                       ovn_stage_get_table(S_SWITCH_IN_L2_LKUP));
     }
-    ds_put_format(&match, REGBIT_NF_ENABLED" == 1 && "
-                          REGBIT_NF_ORIG_DIR" == 1 && "
-                          REG_NF_ID " == %"PRIu8,
-                  (uint8_t) nf->id);
     ovn_lflow_add(lflows, od, fwd_stage, 99, ds_cstr(&match),
                   ds_cstr(&action), lflow_ref);
+
     ds_clear(&match);
     ds_clear(&action);
 
@@ -19876,19 +19910,31 @@ consider_network_function_inline(struct lflow_table *lflows,
     ds_clear(&match);
     ds_clear(&action);
 
-    /* For packets redirected from egress pipleline to the NF, when they come
-     * out from the other NF port, we don't want to process them again through
-     * egress stages they already went through, especially not again through
-     * conntrack as these packets are already accounted for there. Hence we
-     * need to skip the initial pipeline stages for such packets and directly
-     * start from the NF table. The packets that fall under this category are
-     * the response packets from NF for from-lport ACLs and request packets
-     * received from NF for to-lport ACLs. */
+    /* Post-NF Processing: Resumes pipeline after ls_out_nf to bypass
+     * previously completed egress stages (e.g., conntrack). Uses priority
+     * 115 in out_pre_acl to take precedence over priority-110 conntrack
+     * skip flows, and calls nf_lookup_orig_inport() to identify loopback
+     * packets returning on their original inport. */
     ds_put_format(&match, "inport == %s", input_port->json_key);
-    ds_put_format(&action, "next(pipeline=egress, table=%d);",
-                  (ovn_stage_get_table(S_SWITCH_OUT_NF) + 1));
-    ovn_lflow_add(lflows, od, S_SWITCH_OUT_PRE_ACL, 110, ds_cstr(&match),
+    ds_put_format(&action,
+                  REGBIT_NF_LOOKUP_HIT " = nf_lookup_orig_inport(); "
+                  "next(pipeline=egress, table=%d);",
+                  ovn_stage_get_table(S_SWITCH_OUT_NF) + 1);
+    ovn_lflow_add(lflows, od, S_SWITCH_OUT_PRE_ACL, 115, ds_cstr(&match),
+                  ds_cstr(&action), lflow_ref);
+    ds_clear(&match);
+    ds_clear(&action);
+
+    /* Priority 2 in out_nf (output_port): post-NF packet re-entering the
+     * egress pipeline; run the lookup here so ls_out_check_port_sec drops
+     * a loopback copy. */
+    ds_put_format(&match, "inport == %s", output_port->json_key);
+    ds_put_format(&action,
+                  REGBIT_NF_LOOKUP_HIT " = nf_lookup_orig_inport(); next;");
+    ovn_lflow_add(lflows, od, S_SWITCH_OUT_NF, 2, ds_cstr(&match),
                   ds_cstr(&action), lflow_ref);
+    ds_clear(&match);
+    ds_clear(&action);
 
     /* Priority 120 flows in out_stateful:
      * If packet was received on a tunnel interface and being forwarded to a
@@ -20107,6 +20153,7 @@ build_network_function(const struct ovn_datapath *od,
 {
     unsigned long *nfg_ingress_bitmap = bitmap_allocate(MAX_OVN_NF_GROUP_IDS);
     unsigned long *nfg_egress_bitmap = bitmap_allocate(MAX_OVN_NF_GROUP_IDS);
+    bool has_inline_nfg = false;
 
     /* This flow matches packets injected from out_nf stage -
      * after it sets the outport - back to in_l2_lkup stage. This rule must be
@@ -20169,12 +20216,16 @@ build_network_function(const struct ovn_datapath *od,
                            ? true : false;
             unsigned long *nfg_bitmap = ingress ? nfg_ingress_bitmap
                                                 : nfg_egress_bitmap;
-            unsigned int nfg_id = acl->network_function_group->id;
-            if (bitmap_is_set(nfg_bitmap, nfg_id)) {
+            const struct nbrec_network_function_group *nfg =
+                acl->network_function_group;
+            if (bitmap_is_set(nfg_bitmap, nfg->id)) {
                 /* Skip if same NF is used again. */
                 continue;
             }
-            nfg_bitmap = bitmap_set1(nfg_bitmap, nfg_id);
+            nfg_bitmap = bitmap_set1(nfg_bitmap, nfg->id);
+            if (!has_inline_nfg && !network_function_group_is_vtap_mode(nfg)) {
+                has_inline_nfg = true;
+            }
             consider_network_function(lflows, od, acl->network_function_group,
                                       ingress, lflow_ref);
         }
@@ -20194,19 +20245,47 @@ build_network_function(const struct ovn_datapath *od,
                     unsigned long *nfg_bitmap = ingress
                                                 ? nfg_ingress_bitmap
                                                 : nfg_egress_bitmap;
-                    unsigned int nfg_id = acl->network_function_group->id;
-                    if (bitmap_is_set(nfg_bitmap, nfg_id)) {
+                    const struct nbrec_network_function_group *nfg =
+                        acl->network_function_group;
+                    if (bitmap_is_set(nfg_bitmap, nfg->id)) {
                         /* Skip if same NF is used again. */
                         continue;
                     }
-                    nfg_bitmap = bitmap_set1(nfg_bitmap, nfg_id);
+                    nfg_bitmap = bitmap_set1(nfg_bitmap, nfg->id);
+
+                    if (!has_inline_nfg &&
+                            !network_function_group_is_vtap_mode(nfg)) {
+                        has_inline_nfg = true;
+                    }
                     consider_network_function(lflows, od,
                                               acl->network_function_group,
                                               ingress, lflow_ref);
                 }
             }
         }
     }
+
+    if (has_inline_nfg) {
+        /* Drop the loopback copy flagged by nf_lookup_orig_inport(). */
+        ovn_lflow_add(lflows, od, S_SWITCH_OUT_CHECK_PORT_SEC, 110,
+                      REGBIT_NF_LOOKUP_HIT " == 1", debug_drop_action(),
+                      lflow_ref);
+
+        /* Overlay only: always learn the inport so a post-NF loopback copy
+         * can be detected and dropped when redirection happens on a different
+         * node than where the packet originally entered the LS. */
+        if (!ls_has_localnet_port(od)) {
+            ovn_lflow_add(lflows, od, S_SWITCH_IN_NF, 50,
+                          "ip4 && " REGBIT_NF_ENABLED" == 0",
+                          "nf_learn_orig_inport(ipv6 = false); next;",
+                          lflow_ref);
+            ovn_lflow_add(lflows, od, S_SWITCH_IN_NF, 50,
+                          "ip6 && " REGBIT_NF_ENABLED" == 0",
+                          "nf_learn_orig_inport(ipv6 = true); next;",
+                          lflow_ref);
+        }
+    }
+
     bitmap_free(nfg_ingress_bitmap);
     bitmap_free(nfg_egress_bitmap);
 }

ovn-nb.xml

@@ -6760,6 +6760,19 @@ or
       as down, the fallback method is applied.
     </p>
 
+    <p>
+      <b>Known limitation (inline mode)</b>: in topologies where a
+      post-NF flood copy may reach the same logical inport from a
+      different node than the packet's original ingress node,
+      the lookup runs on a different node than where the inport
+      was learned, so loopback copies are not identified.
+      For example: OVN-IC L2 stretch with the NF and IC gateway on
+      different chassis.  With <code>unknown</code>-addressed ports, the
+      post-NF packet can be flooded across AZs and come back to the
+      source AZ via another node, so the lookup does not hit the
+      original learn entry.
+    </p>
+
     <column name="name">
       Name of the <ref table="Network_Function_Group"/>. Name should be unique.
     </column>