Currently, the ktrans-ping container appears to initiate a distinct HTTP request to the New Relic ingest API for each individual ping result. In high-scale environments where multiple nodes are monitored with high frequency, this behavior results in an extremely high connection velocity from a single source IP.
This high volume of Layer 7 requests is being flagged as "aggressive" or potential "DDoS-like" activity by edge security layers (e.g., Cloudflare), leading to HTTP 503 Service Unavailable errors before the data even reaches the New Relic application layer (which explains why no "Rate Limit" violations are seen in the New Relic dashboard).
Current Behavior
Ktranslate sends data to New Relic immediately or with very high frequency per ping event.
Each event (or small group of events) triggers a new HTTP connection/request.
Results in high overhead and triggers edge-level security blocking at the ingest gateway.
Desired Behavior (Proposed Solution)
To improve scalability and prevent edge-side blocking, we propose implementing an Internal Buffering and Batching mechanism:
Metric Batching: Allow Ktranslate to cache/buffer ping results locally for a configurable interval (e.g., 20s, 30s, or 60s).
Summarization/Bulk Upload: Instead of sending individual metrics, wrap multiple ping results into a single JSON payload (batch) per HTTP POST request.
Connection Reuse (Keep-Alive): Ensure the underlying HTTP client utilizes persistent TCP connections to reduce the handshake overhead that often triggers security flags.
# Hypothetical configuration for batching
ingest:
batch_window_sec: 30 # Buffer data for 30 seconds before sending
max_batch_size_kb: 1024 # Or send if buffer reaches 1MB
keep_alive: true # Reuse TCP connections
Impact
Reduced Connection Velocity: Dramatically lowers the number of HTTP requests sent to the API.
Improved Reliability: Prevents 503 errors caused by edge-level throttling.
Efficiency: Reduces CPU and Network overhead on the host running the Ktranslate container.
Currently, the ktrans-ping container appears to initiate a distinct HTTP request to the New Relic ingest API for each individual ping result. In high-scale environments where multiple nodes are monitored with high frequency, this behavior results in an extremely high connection velocity from a single source IP.
This high volume of Layer 7 requests is being flagged as "aggressive" or potential "DDoS-like" activity by edge security layers (e.g., Cloudflare), leading to HTTP 503 Service Unavailable errors before the data even reaches the New Relic application layer (which explains why no "Rate Limit" violations are seen in the New Relic dashboard).
Current Behavior
Ktranslate sends data to New Relic immediately or with very high frequency per ping event.
Each event (or small group of events) triggers a new HTTP connection/request.
Results in high overhead and triggers edge-level security blocking at the ingest gateway.
Desired Behavior (Proposed Solution)
To improve scalability and prevent edge-side blocking, we propose implementing an Internal Buffering and Batching mechanism:
Metric Batching: Allow Ktranslate to cache/buffer ping results locally for a configurable interval (e.g., 20s, 30s, or 60s).
Summarization/Bulk Upload: Instead of sending individual metrics, wrap multiple ping results into a single JSON payload (batch) per HTTP POST request.
Connection Reuse (Keep-Alive): Ensure the underlying HTTP client utilizes persistent TCP connections to reduce the handshake overhead that often triggers security flags.
Impact
Reduced Connection Velocity: Dramatically lowers the number of HTTP requests sent to the API.
Improved Reliability: Prevents 503 errors caused by edge-level throttling.
Efficiency: Reduces CPU and Network overhead on the host running the Ktranslate container.