[Bug]: bgpd: static passive peers fail open when ttl-security MinTTL setup fails on accepted socket

[xinzheliu2000]

Summary

FRR bgpd appears to fail open for configured static incoming peers when GTSM
socket setup fails.

For a static passive peer configured with ttl-security hops, if
bgp_set_socket_ttl() fails while handling the accepted incoming socket, FRR
logs the error at debug level and continues. In dynamic testing, the peer still
reached Established even though MinTTL enforcement failed.

I am reporting this as a GTSM enforcement / security hardening issue, not as a
remote packet parser vulnerability.

I do not claim that a remote peer can normally cause the socket option failure.
The issue is that once the configured enforcement cannot be installed, the
static incoming path continues while other paths fail closed.

Test Configuration

router bgp 65001
 bgp router-id 127.0.0.1
 no bgp ebgp-requires-policy
 neighbor 127.0.0.2 remote-as 65002
 neighbor 127.0.0.2 passive
 neighbor 127.0.0.2 ttl-security hops 1
 !
 address-family ipv4 unicast
  neighbor 127.0.0.2 activate
 exit-address-family

Expected Behavior

If ttl-security hops is configured and FRR cannot apply the required
MinTTL/HopLimit check to an accepted incoming socket, the incoming connection
should be rejected or the peer should be reset.

At minimum, the session should not reach Established while silently lacking
the configured inbound GTSM enforcement.

Actual Behavior

For a static incoming peer, FRR logs the failed TTL setup but continues:

BGP: [Event] Unable to set min/max TTL on peer 127.0.0.2, Continuing

The same peer then reaches Established:

BGP state = Established, up for 00:00:01

Dynamic Verification

This was reproduced against:

FRR bgpd 10.6.1

The PoC uses LD_PRELOAD to make setsockopt(IP_MINTTL) and
setsockopt(IPV6_MINHOPCOUNT) fail inside bgpd, simulating a local/platform
failure to install inbound GTSM enforcement.

This is a simulated local/platform socket-option failure. It demonstrates the
control-flow behavior after MinTTL setup fails; it is not a claim that the BGP
peer can remotely force setsockopt(IP_MINTTL) to fail.

PoC output:

=== FRR static incoming GTSM MinTTL failure verification ===
preload: /tmp/frr_gtsm_failopen_.../fail_minttl.so
[+] TCP connected ('127.0.0.2', ...) -> ('127.0.0.1', 11921)
[*] Sent BGP OPEN
[*] Received pre-test message: type=1 body_len=64
[*] Sent KEEPALIVE
[*] Received pre-test message: type=4 body_len=0
...
BGP state = Established, up for 00:00:01
...
BGP: [Event] Unable to set min/max TTL on peer 127.0.0.2, Continuing
...
VERDICT: FAIL_OPEN_REPRODUCED

Source Analysis

bgp_set_socket_ttl() returns an error when GTSM MinTTL setup fails:

ret = sockopt_minttl(connection->su.sa.sa_family, connection->fd,
                     MAXTTL + 1 - peer->gtsm_hops);
if (ret) {
  flog_err(EC_LIB_SOCKET,
           "%s: Can't set MinTTL on peer (rtrid %pI4) socket, err = %d",
           __func__, &peer->remote_id, errno);
  return ret;
}

The dynamic incoming peer path fails closed:

if (bgp_set_socket_ttl(incoming) < 0) {
  ...
  incoming->fd = -1;
  close(bgp_sock);
  return;
}

However, the static incoming peer path continues after the same failure:

if (bgp_set_socket_ttl(incoming) < 0)
  if (bgp_debug_neighbor_events(doppelganger))
    zlog_debug("[Event] Unable to set min/max TTL on peer %s, Continuing",
               doppelganger->host);

This creates inconsistent behavior between dynamic and static incoming peers.
Outgoing connections also fail closed when bgp_set_socket_ttl() fails.

In other words, FRR already treats this failure as fatal in dynamic incoming
and outgoing paths. Only the static incoming doppelganger path logs the failure
and proceeds to FSM startup.

Impact

The impact is an operator-visible enforcement gap. A peer configured with
ttl-security hops may establish as a static incoming session even though the
kernel did not install the inbound MinTTL/HopLimit check for that accepted
socket.

This does not by itself give an attacker the ability to establish an
unauthorized BGP session. The peer must still match normal BGP peer
configuration and complete the TCP/BGP handshake. The risk is that GTSM may be
assumed to be active when it was not actually enforced on the incoming socket.

The peer cannot normally trigger the socket-option failure remotely. The
behavior is remotely observable once such a local/platform failure occurs, but
the failure itself is not caused by BGP UPDATEs or other BGP payloads.

Suggested Fix Direction

Handle the static incoming peer path like the dynamic incoming peer path. The
exact cleanup should match FRR's doppelganger peer lifecycle, but the accepted
socket should not proceed to FSM startup after GTSM setup fails.

For example, the error path should be equivalent in spirit to:

if (bgp_set_socket_ttl(incoming) < 0) {
  peer_set_last_reset(doppelganger, PEER_DOWN_SOCKET_ERROR);
  incoming->fd = -1;
  close(bgp_sock);
  return;
}

If there is a compatibility reason to keep the current behavior, it would be
useful to document that ttl-security hops may fail open for static incoming
peers when MinTTL/HopLimit socket setup is unavailable.