[BUG] NPM Gateway Lookup Incorrectly Reports Private Intra-VPC Traffic as NAT Gateway Traffic When AWS_VPC_K8S_CNI_EXTERNALSNAT=true

[jangeja]

Agent version

7.x (affects all versions with NPM gateway lookup feature enabled)

Bug Report

When AWS_VPC_K8S_CNI_EXTERNALSNAT=true is configured in AWS EKS environments with nftables-based kube-proxy, the Network Performance Monitoring (NPM) gateway lookup feature incorrectly classifies private intra-VPC traffic (e.g., pod-to-Kubernetes-service communication) as traffic going through an AWS NAT gateway.

Expected Behavior:

• Private network traffic between pods and Kubernetes services (using private service IPs like 10.x.x.x or 172.x.x.x) should be reported as direct intra-VPC routing
• Gateway type should only be reported as "NAT gateway" for traffic actually egressing through a NAT gateway to the internet

Actual Behavior:

• All traffic with a routing table gateway entry is reported with subnet information
• Backend interprets this as NAT gateway traffic, even when:
  • Destination is a private Kubernetes service (ClusterIP)
  • Traffic never leaves the VPC
  • Hostnames resolve to private IPs within the cluster

Root Cause

The gateway lookup implementation in pkg/network/gateway_lookup_linux.go lacks any logic to differentiate between:

1. Traffic egressing through an actual NAT gateway to the internet
2. Private intra-VPC traffic that happens to have a gateway entry in the routing table

Code Analysis:

In pkg/network/gateway_lookup_linux.go:127-137, the LookupWithIPs() method only checks if a gateway exists:

func (g *gatewayLookup) LookupWithIPs(source util.Address, dest util.Address, netns uint32) *Via {
    r, ok := g.routeCache.Get(source, dest, netns)
    if !ok {
        return nil
    }

    // if there is no gateway, we don't need to add subnet info
    // for gateway resolution in the backend
    if !r.Gateway.IsValid() || r.Gateway.IsUnspecified() {
        return nil
    }
    
    // ... proceeds to look up subnet via EC2 metadata
}

Missing checks:

• Whether destination IP is private (RFC1918: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16)
• Whether destination is within the same VPC CIDR as source
• Whether traffic is actually leaving the VPC

Reproduction Steps

1. Deploy agent in AWS EKS cluster with nodes in private subnets
2. Configure AWS VPC CNI with AWS_VPC_K8S_CNI_EXTERNALSNAT=true (enables nftables for kube-proxy)
3. Enable NPM gateway lookup feature (enabled by default as of recent versions)
4. Generate traffic from pods to Kubernetes services (ClusterIP type)
5. Observe network metrics

Result: Private service traffic shows server_gateway_type: aws_nat_gateway even though server_domain shows private Kubernetes service names (e.g., my-service.default.svc.cluster.local)

Why This Happens with AWS_VPC_K8S_CNI_EXTERNALSNAT=true

When AWS_VPC_K8S_CNI_EXTERNALSNAT=true:

• AWS VPC CNI disables SNAT iptables rules (as documented in AWS EKS External SNAT docs)
• kube-proxy configures service routing via nftables
• Routing table entries for Kubernetes service traffic now include gateway IPs
• Agent's gateway lookup sees these gateways and queries EC2 metadata for subnet information
• Backend receives: {gateway: present, subnet_id: subnet-xxx} → incorrectly infers NAT gateway

Proposed Fix

Add destination type checking in pkg/network/gateway_lookup_linux.go before querying EC2 metadata and returning subnet information:

func (g *gatewayLookup) LookupWithIPs(source util.Address, dest util.Address, netns uint32) *Via {
    r, ok := g.routeCache.Get(source, dest, netns)
    if !ok {
        return nil
    }

    if !r.Gateway.IsValid() || r.Gateway.IsUnspecified() {
        return nil
    }
    
    // NEW: Don't report gateway info for private intra-VPC traffic
    // If both source and destination are private IPs, this is likely
    // intra-VPC traffic and should not be classified as NAT gateway traffic
    if dest.Addr().IsPrivate() && source.Addr().IsPrivate() {
        return nil
    }
    
    // ... rest of existing subnet lookup logic
}

Alternatively, a more sophisticated check could:

1. Query VPC CIDR ranges from EC2 metadata
2. Check if both source and destination fall within the same VPC CIDR
3. Only report gateway/subnet info for traffic destined outside the VPC

Agent configuration

# Standard NPM configuration
network_config:
  enabled: true
  enable_gateway_lookup: true  # Enabled by default in recent versions

Operating System

Linux (kernel with netlink support, tested on Amazon Linux 2)

Other environment details

• Cloud Provider: AWS EKS
• VPC CNI Configuration: AWS_VPC_K8S_CNI_EXTERNALSNAT=true
• kube-proxy mode: nftables
• Network setup: Nodes in private subnets with NAT gateway for internet egress
• Affected traffic: Pod-to-Service communication (ClusterIP services with private IPs)

Related Code Files

• pkg/network/gateway_lookup_linux.go - Main gateway lookup logic
• pkg/network/route_cache.go - Routing table query via netlink
• pkg/util/ec2/network.go - EC2 metadata API subnet lookup
• pkg/network/tracer/tracer.go:778 - Gateway lookup invocation