[iris] Warn on log push RPC failures instead of silently swallowing

[claude]

RPC failures in LogPusher._send() and the task_attempt log helpers were logged at
DEBUG level, making log-server outages completely invisible to operators. Changed
to WARNING with rate limiting (warn on first failure, then at most every 60s) so
persistent outages don't spam. Logs an INFO recovery message when sends resume
after a failure streak. Also promotes the three debug-level catches in
task_attempt.py (_push_logs, _stream_logs, _cleanup) to warning.

Fixes #4707

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 995dce86a3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Avoid re-entrant warning logs while holding _send_lock

In LogPusher._send, the new logger.warning(...) runs while _send_lock is held. In worker mode, RemoteLogHandler is attached to the root logger, so this warning is fed back into the same LogPusher.push() path; if that handler key buffer is at batch_size - 1 (or batch_size=1), push() immediately re-enters _send() and blocks on the same non-reentrant lock. This creates a deadlock path during log-server RPC failures, which is exactly when this branch executes.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Use Iris timing primitives instead of raw time.monotonic

lib/iris/AGENTS.md (scoped to this file) requires Iris code to use rigging.timing utilities for time-related logic, but this commit introduces direct time.monotonic() usage for warning throttling. This is an explicit project-rule violation in the touched path and should be converted to the shared timing abstractions used elsewhere in Iris.

Useful? React with 👍 / 👎.

[rjpower]

@claude fix the issues

[claude]

Claude finished @rjpower's task in 5m 17s —— View job

---

Fixing Codex Review Issues

• Gather context: read changed files and Iris timing primitives
• Fix P1: Avoid re-entrant warning logs while holding _send_lock (deadlock risk)
• Fix P1: Use rigging.timing.RateLimiter instead of raw time.monotonic
• Update tests to match new implementation
• Run pre-commit and tests
• Commit and push

Summary

Fixed both P1 issues from the Codex review:

1. Deadlock prevention — logger.warning() and logger.info() in LogPusher._send() were called while holding _send_lock. Since RemoteLogHandler is attached to the root logger, this warning could re-enter push() → _send() and block on the same non-reentrant lock. Fix: capture exception info and failure counts inside the lock, then log after releasing it using try/except/else control flow.

2. Use rigging.timing.RateLimiter — Replaced raw time.monotonic() + _last_fail_warn_time with rigging.timing.RateLimiter(interval_seconds=_WARN_INTERVAL), per Iris's coding convention to use rigging.timing for all time-related operations. The RateLimiter.should_run() naturally handles both first-failure (returns True when no prior run) and rate-limited subsequent failures.

Also cleaned up dead test code (unused FailingPusher subclass and sent_count variable).

Testing: ./infra/pre-commit.py --all-files --fix passes. All 9 tests in lib/iris/tests/test_remote_log_handler.py pass ✅
·

[rjpower]

@claude can we not use a special logger for the log pusher itself which bypasses the re-entrant issue? that seems safer than this dance.

If we fail to push for a batch, can we retain the batch to try pushing again later?

[claude]

Claude finished @rjpower's task in 8m 16s —— View job

---

Implementing LogPusher improvements

• Read current implementation (client.py, task_attempt.py, tests)
• Use a dedicated non-propagating logger for LogPusher diagnostics
• Retain failed batches for retry on next flush (with buffer cap)
• Update tests
• Run pre-commit and tests
• Commit and push

Summary

Implemented both suggestions:

1. Dedicated non-propagating logger — Created _pusher_logger (iris.log_server.client._internal) with propagate=False and a StreamHandler to stderr. This structurally prevents the re-entrant deadlock: since _pusher_logger never reaches the root logger, a RemoteLogHandler attached there can't feed warnings back into push() → _send(). This replaces the warn_args/warn_exc_info dance of capturing state inside the lock and logging outside it — now _send() just logs directly.

2. Retry failed batches — _send() now calls _requeue(key, entries) on failure, putting entries back into the buffer for the next flush cycle. Buffer is capped at _MAX_BUFFER_PER_KEY = 10,000 entries per key to prevent unbounded memory growth during extended outages; oldest entries are dropped when the cap is exceeded.

Testing: ./infra/pre-commit.py --all-files --fix passes ✅ | All 11 tests in test_remote_log_handler.py pass ✅ (including 2 new tests: test_failed_entries_retried_on_next_flush and test_requeue_caps_buffer_size)
· Branch