fix(llmobs): emit a single error log when payload send retries are exhausted

[shinsuke938]

Description

Fixes #17476.

LLMObs retries payload delivery up to three times after connection failures. Previously,
every failed attempt emitted an ERROR log with a traceback. If all attempts failed,
periodic() emitted the same ERROR again when the payload was finally dropped.
As a result:

• A transient failure still produced an ERROR, even when a later retry succeeded.
• An exhausted retry sequence could invoke the same ERROR log four times for one batch.
• The logs did not distinguish a failed attempt from a definitively dropped payload.
  This change logs individual connection failures at DEBUG level and describes them as
  retry attempts. The exception is included using %r, which preserves its type and message
  without repeating the full traceback.
  The existing ERROR in periodic() remains the final failure signal. It is emitted only
  after all retry attempts fail and retains exc_info=True, so one traceback is available
  for the dropped payload.
  Retry behavior and dropped-payload telemetry are unchanged.

Testing

Added regression tests covering both connection-failure outcomes:

• A transient failure followed by a successful retry emits one retry DEBUG, no ERROR,
  and no dropped-payload telemetry.
• Three failed attempts emit three retry DEBUG logs, one final ERROR, and one
  dropped-payload telemetry record.

Risks

This intentionally reduces ERROR volume for connection failures. Monitors that rely on
each failed attempt being logged at ERROR will no longer observe transient failures;
per-attempt details now require DEBUG logging.
The final failed to send ... LLMObs ... events error message is unchanged, so queries
matching that message will continue to detect payloads dropped after retries are exhausted.

Additional Notes

HTTP error responses (resp.status >= 300) retain their existing ERROR logging and
dropped-payload telemetry behavior. This change only affects connection exceptions handled
by the retry path.

[datadog-datadog-us1-prod]

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 2f12c38 | Docs | Datadog PR Page | Give us feedback!}

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-08-14 06:27:32

Comparing candidate commit 2f12c38 in PR branch shinsuke/fix-llmobs-retry-logs with baseline commit 6263d5f in branch main.

Found 0 performance improvements and 6 performance regressions! Performance is the same for 615 metrics, 9 unstable metrics.

scenario:httppropagationinject-ids_only

• 🟥 execution_time [+2.495µs; +2.686µs] or [+11.718%; +12.615%]

scenario:iastaspectsospath-ospathbasename_aspect

• 🟥 execution_time [+94.283µs; +100.635µs] or [+22.901%; +24.444%]

scenario:iastaspectssplit-rsplit_aspect

• 🟥 execution_time [+10.799µs; +14.514µs] or [+7.738%; +10.400%]

scenario:span-start

• 🟥 execution_time [+1.342ms; +1.536ms] or [+8.315%; +9.516%]

scenario:telemetryaddmetric-1-count-metric-1-times

• 🟥 execution_time [+458.698ns; +517.213ns] or [+17.110%; +19.293%]

scenario:tracer-small

• 🟥 execution_time [+29.599µs; +31.471µs] or [+8.476%; +9.012%]

[shinsuke938]

This ERROR log predates the retry path. It originated in a single-attempt writer
where the except block was inline in periodic() with no raise and no retry
wrapper (see 76b8817 and earlier), so an error-level log was the correct signal for
a definitively dropped payload.
When retry logic was added in v2.11.0, the log moved
into the retried _send_payload and a second error log appeared at the periodic()level.
The per-attempt error level was carried over, not chosen for the retry path.