Agent nodes flap between active/inactive due to aggressive health check timeout

[williamhdean]

Description

Agent nodes intermittently disappear from the active node list due to an aggressive health check timeout in the control plane. Nodes sending heartbeats every 2 seconds still experience flapping, where they appear and disappear from the /api/v1/nodes endpoint.

Environment

• Control Plane Version: v0.1.39 (also tested with latest)
• Agent SDK: agentfield==0.1.38 (Python)
• Deployment: Docker containers on same network
• Heartbeat Interval: 2 seconds (default), also tested with 1 second

Steps to Reproduce

1. Start the AgentField control plane via Docker
2. Start 2+ agent nodes that register with the control plane
3. Query the nodes endpoint repeatedly:

   for i in $(seq 1 10); do
     curl -s "http://localhost:8080/api/v1/nodes" | jq -c '[.nodes[].id]'
     sleep 2
   done

Expected Behavior

All registered nodes should consistently appear in the node list as long as they are sending heartbeats within the configured interval.

Actual Behavior

Nodes intermittently disappear from the list. Example output:

["campaign-planner", "content-writer"]
["campaign-planner"]
[]
["content-writer"]
["campaign-planner", "content-writer"]
["campaign-planner"]

The control plane logs show:

{"level":"error","error":"agent node with ID 'content-writer' not found","message":"❌ Node content-writer not found during heartbeat update"}

Workarounds

1. Query specific nodes directly: GET /api/v1/nodes/{node_id}
2. Query without health filter: GET /api/v1/nodes?health_status=

Feature Request

Please add configurable health check timeout settings, either via:

1. Environment variables:

   AGENTFIELD_NODE_HEALTH_TIMEOUT=30s
   AGENTFIELD_HEARTBEAT_TIMEOUT=10s
   

2. YAML configuration:

   nodes:
     health_timeout: 30s
     heartbeat_timeout: 10s

The current ~2-3 second timeout is too aggressive for production use, especially in containerized environments where network latency can vary.

Additional Context

• Tested with both SQLite (local) and attempted PostgreSQL storage modes
• Issue persists across control plane restarts
• Agents successfully re-register on next heartbeat, but the flapping causes issues for monitoring and orchestration

[santoshkumarradha]

@AbirAbbas let's pick this backlog as well.

[github-actions]

Thanks for picking this up, @AbirAbbas! 🎉

Here's what to expect:

• We'll check in after 7 days if we haven't heard an update
• After 14 days, we'll send a second reminder
• After 21 days without activity, we'll unassign to let others contribute

Tips for success:

• Drop a comment when you start working on it
• Share any blockers - we're here to help!
• Link your PR with Fixes #167 in the description

Happy coding! 🚀

[AbirAbbas]

#169