490 baseline loadtest saw 1.1k errors and one of the ECS containers restarted due to out of memory

[AndreyKizimenko]

Fleet versions

• Discovered: 4.90
• Reproduced: 4.90

Web browser and operating system:

---

💥  Actual behavior

988 occurrences:

{"ts":"2026-07-30T02:46:34Z","level":"error","component":"http","method":"POST","uri":"/api/osquery/distributed/read","took":5022711880,"host_id":17725,"ip_addr":"10.12.3.184","x_for_ip_addr":"10.12.3.184","err":"dial tcp 10.12.31.69:6379: i/o timeout"}

75:

{"ts":"2026-07-30T02:46:34Z","level":"error","component":"http","method":"POST","uri":"/api/osquery/log","took":5016577014,"host_id":16929,"ip_addr":"10.12.2.236","x_for_ip_addr":"10.12.2.236","results":8,"err":"dial tcp 10.12.31.69:6379: i/o timeout"}

2:

{"ts":"2026-07-30T02:46:34Z","level":"error","msg":"QueriesForHost","err":"load active queries: load cache: get active queries: dial tcp 10.12.31.69:6379: i/o timeout"}

1:

{"ts":"2026-07-30T02:46:34Z","level":"error","msg":"QueriesForHost","err":"flush pipeline: dial tcp 10.12.31.69:6379: i/o timeout"}

🛠️ Expected behavior

TODO

🧑‍💻  Steps to reproduce

These steps:

• Have been confirmed to consistently lead to reproduction in multiple Fleet instances.
• Describe the workflow that led to the error, but have not yet been reproduced in multiple Fleet instances.

1. Review 4.90 baseline loadtest instance
2. https://us-east-2.console.aws.amazon.com/cloudwatch/home?region=us-east-2#logsV2:logs-insights$3FqueryDetail$3D~(end~0~start~-86400~timeType~'RELATIVE~tz~'UTC~unit~'seconds~editorString~'fields*20*40timestamp*2c*20*40message*2c*20*40logStream*2c*20*40log*0a*7c*20filter*20ispresent*28error*29*20or*20ispresent*28err*29*20or*20level*20*3d*20*27error*27*0a*7c*20filter*20*40message*20not*20like*20*2ffleet_detail_query_software*2f*0a*7c*20filter*20level*20*21*3d*20*27debug*27*0a*7c*20filter*20level*20*21*3d*20*27info*27*0a*7c*20sort*20*40timestamp*20desc*0a*7c*20limit*2010000*0a~queryId~'264cdc24-afab-4d53-a44e-2eaf39dba6ad~source~(~'fleet-490loadtest)~lang~'CWLI~logClass~'STANDARD~queryBy~'logGroupName)$26tab$3DlogPatterns

🕯️ More info (optional)

Slack thread: https://fleetdm.slack.com/archives/C051QJU3D0V/p1785443180642689