Increased latency and upstream/client timeouts (504s observed at API Gateway, 499/408 in NIC logs) after migration from community ingress-nginx to F5 NIC OSS

[Santosh-k8s]

Version

• F5 NGINX Ingress Controller version: 5.4.2
• Kubernetes version: 1.34.3
• Platform: Anthos on-prem
• Install method: Helm
• Number of NIC pods / resources: 10 pods with cpu request of 10 on 24cpu dedicated nodes

Steps to reproduce

1. Migrate application ingress objects from community ingress-nginx to F5 NGINX Ingress Controller OSS.
2. Keep the upstream application and external API Gateway path unchanged.
3. Send production-like traffic through the existing flow:
   • Client -> API Gateway -> F5/GTM/LB -> F5 NIC -> backend service/pods
4. Observe request latency, API Gateway 504s, and NGINX access/error logs during the same time window.

Expected behavior

After migration from community ingress-nginx to F5 NIC OSS, application behavior should remain functionally equivalent under similar traffic patterns:

• no noticeable latency regression
• no increase in API Gateway 504s
• no abnormal increase in NGINX 499/408 log entries compared to the previous controller

Actual behavior

After migrating from community ingress-nginx to F5 NIC OSS, handful of applications exposed through our API Gateway are only showing:
(API gateway has 9s upstream read timeout )

• increased end-to-end latency
• increased client-facing/API Gateway 504 Gateway Timeout responses
• corresponding NGINX log entries showing 499 OR 408

This behavior was not observed at the same level before the migration on the community ingress controller.

Traffic topology

Client -> API Gateway -> external load balancer / GTM -> F5 NIC OSS -> Kubernetes Service -> backend pods

Symptoms observed

• Client/API Gateway sees more 504 responses after the migration.
• F5 NIC access logs show 499 and 408 for affected requests.
• The issue is intermittent but frequent enough to cause user-visible impact.
• The problem appears under real application traffic behind the API Gateway rather than in a simple direct curl-only path.
• Noticed when traffic / volume is high

What changed

The primary change was the ingress controller migration:

• Before: community ingress-nginx
• After: F5 NGINX Ingress Controller OSS

The backend application, exposure path, and API Gateway integration remained broadly the same.

Why we are opening this issue

We want to understand whether there are known behavioral differences between community ingress-nginx and F5 NIC OSS around request handling, buffering, timeout propagation, upstream connection reuse, or client/proxy disconnect handling that could explain:

• higher latency after migration
• more API Gateway 504s
• more 499 / 408 entries in NIC logs

Troubleshooting already done

We have already:

• compared behavior before vs after migration
• verified that the issue correlates with the controller change window
• confirmed the external symptom (504) and the NIC-side log symptom (499 / 408)
• observed the issue across real application traffic paths behind the API Gateway

Questions for maintainers

1. Are there any known defaults or behavioral differences in F5 NIC OSS vs community ingress-nginx that could increase:
   • request latency
   • likelihood of upstream/client timeout mismatches
   • 499 / 408 occurrences
2. Are there specific timeout, buffering, keepalive, or request-body handling settings we should review first for parity with community ingress-nginx?
3. Is there a recommended logging format or diagnostic data set you want us to provide to help isolate whether the disconnect is:
   • client/API Gateway side
   • external LB/GTM side
   • NIC side
   • upstream application side

Impact

This is causing user-visible latency degradation and increased timeout failures for migrated applications, so we would appreciate guidance on:

• parity settings to review first
• recommended diagnostics
• any known migration considerations between community ingress-nginx and F5 NIC OSS for timeout-sensitive traffic behind an API Gateway