[PERFORMANCE] 100% cron-session failure rate from DeepSeek provider connection drops (ReadError/APIConnectionError) aborts evolution pipeline

[Da-Mikey]

Problem (observed in real usage)

100% of cron-scheduled Hermes sessions in the last 7 days (18 of 18) terminated with provider-level connection errors before completing their scheduled task. 61% of these (11/18) died with ≤2 messages — the model never even emitted a single tool call.

Evidence (aggregated, anonymized)

• Scope: All 18 cron sessions recorded in the last 7 days (2026-06-20 through 2026-06-27)
• Error types: ReadError (12/18, broken pipe mid-stream) and APIConnectionError (6/18, timeout on connect)
• Model: All sessions pinned to deepseek-v4-flash
• Jobs affected (by cron job ID): 0b013aa0bd43 (runs 4-8×/day — evolution introspection/analysis), 332261ca5226 (oracular dream — 2× failed), 3fe172c3c437, c0d29b70d1f4, e0aa9a3d6252, e2864ae22932, 39feda54b944
• Session dump files (under ~/.hermes/sessions/, all end with error objects): all 18 files present with failure_category: timeout in error objects
• Comparison to user sessions: User sessions in the same period show far lower failure rates — the DeepSeek API appears to drop connections at a higher rate for short-lived cron sessions than for longer user interactions

Impact on real tasks

The entire evolution self-improvement pipeline runs as cron jobs. Every single evolution stage (introspection, research, issues, analysis, implementation, integration, upstream-sync, hydra) depends on cron sessions completing. When every cron job fails at the provider connection layer, the pipeline produces no output:

• Introspection: cannot analyze sessions or file issues
• Research: never written
• Issues: never filed
• Implementation: never started
• Integration: never attempted

This is a complete pipeline stall. The system is failing silently — the cron architecture logs error but there is no health check or alerting that surfaces a 100% failure rate.

Proposed direction

1. Diagnose the root cause: Does the DeepSeek API endpoint drop connections for short-lived sessions? Is there a connection-pool exhaustion or TLS renegotiation issue in the Hermes cron runner (e.g. no keepalive for frequent cron sessions)?

2. Add provider-level retry with exponential backoff for cron sessions: The current retry logic may not cover ReadError (broken pipe mid-response) — these are treated as terminal when they may be transient.

3. Add cron health monitoring: A simple script that checks whether the last N cron sessions completed with actual tool calls and non-error status. Surface the result to the evolution-health pipeline so a 100% failure rate is visible.

4. Consider a provider fallback per-cron: If deepseek-v4-flash is dropping connections, allow cron jobs to fall back to another configured provider/model before failing.

Value

• Impact: 1.0 (task impossible — entire evolution pipeline blocked)
• Effort: 0.5 (diagnosis + retry path + monitoring)
• Priority Score: 1.0 × 2 / 0.5 = 4.0

[Lexus2016]

Integration review — blocked: dead code / not wired.

PR #620 added , which correctly reads cron failure records and builds a report, but the script has no caller: there is no job to schedule it, and no existing evolution stage invokes it. Until it is wired into the cron scheduler, the 100% failure-rate alerting described in this issue cannot actually fire.

Done criteria for the next attempt:

1. Add with , , and a schedule (e.g. every 6 hours) so creates a native cron job.
2. Confirm is invoked by the scheduler from and writes .
3. Optional but encouraged: wire when unhealthy so an alert issue is opened automatically.

Re-open a PR once the script is reachable from the evolution cron registry.

[Lexus2016]

Blocked at implementation: the referenced PR #620 script (scripts/cron_health_check.py) is not present in current main.\n\nVerification performed in this session:\n- find . -name "cron_health_check.py" → 0 matches\n- grep -rni "cron_health_check" --include=*.py → 0 matches\n- gh pr view 620 --json files confirms the PR added scripts/cron_health_check.py, but the file is absent from main (likely not merged or reverted).\n\nNeed either (a) the exact path/contents of the health script to wire, or (b) PR #620 merged into main first. Keeping open with blocked label.

[Lexus2016]

Blocked this cycle. The cron health script referenced in PR #620 is not present in current main: searches for evolution_health.py, cron digest, and failure category returned no matching script. Please provide the exact script name/path in PR #620 or a direct link to the file so the wiring/integration step can be verified.

[Lexus2016]

Verified abated — closing. Live pipeline data on the evolution server (2026-07-01) no longer shows the failure mode this issue describes.

Current cron health (since #645 deployed, 2026-06-30 13:00):

• 139 cron runs, 1 failure → 99.3% success. The single failure was evolution-implementation self-reporting RuntimeError: agent reported failure (an agent-level task outcome, not a provider drop).
• All agent stages completing today — upstream-sync, implementation, integration, introspection, analysis — multiple runs/day.
• Zero ReadError / APIConnectionError DeepSeek connection drops in the last 3 days. The acute provider-side drop storm (2026-06-20…27) was transient and has cleared.

On the bulk recent failures: the only high-count failure in the window was evolution-ci-diagnosis (35×, "GITHUB_TOKEN required") — a different root cause (the subprocess sanitizer strips GITHUB_TOKEN from no_agent scripts), already fixed by #645 (gh auth token fallback). Green since 2026-06-30 12:30.

Durable protection against provider-side drops (per-entry failure tracking + known-bad-provider disable threshold) is tracked in #572 — the right long-term fix, not the session-monitoring approach in the now-closed #620 (dead code, never wired into the scheduler).

Closing as no longer reproducible. Reopen if the 100%-failure signature returns.