Skip to content

feat: add retry support for SQL/HTTP activities and loop resilience - #44

Closed
Pino de Candia (pinodeca) wants to merge 1 commit into
mainfrom
pinodeca/retries
Closed

feat: add retry support for SQL/HTTP activities and loop resilience#44
Pino de Candia (pinodeca) wants to merge 1 commit into
mainfrom
pinodeca/retries

Conversation

@pinodeca

Copy link
Copy Markdown
Contributor

Summary

Implements the retry support proposal (Phases 1 & 2): automatic retries for SQL/HTTP activities and loop resilience for long-running loops.

Changes

Phase 1: Node Retries

  • New GUC: pg_durable.max_retries (default 3, reloadable via pg_reload_conf())
  • SQL and HTTP activities use schedule_activity_with_retry with exponential backoff (100ms base, 2x multiplier, 30s cap)
  • Status update activities (update-node-status, update-instance-status) use fixed 1s backoff retry
  • max_retries is captured at df.start() time and stored in FunctionInput for deterministic replay

Phase 2: Loop Resilience

  • Failed loop iterations are absorbed — the loop logs the error and continues to the next iteration
  • LoopMetrics struct tracks total iterations, successes, failures, and consecutive failures across continue_as_new boundaries
  • Safety limit: 10 consecutive failures terminate the loop (protects against permanently broken queries)
  • Trace logging per iteration with rolling-window summary

Documentation

  • USER_GUIDE.md: New Configuration section with GUC reference, Loop Resilience subsection, updated Key Features and Quick Reference
  • docs/api-reference.md: Retry notes on df.sql(), df.http(), df.loop()

Tests

  • 36_retry_node.sql: Verifies SQL node succeeds after transient failures (attempt 3/3), and fails after exhausting retries
  • 37_loop_resilience.sql: Verifies loop continues past intermittent failures, and terminates after consecutive failure limit

Design Decisions

Decision Choice Rationale
Retry scope Global GUC, not per-node Simpler v1; per-node deferred to Phase 3
GUC capture At df.start() time Deterministic — same value used across replays
Loop resilience Always on Loops are long-running supervisors by design
Consecutive failure limit 10 (hardcoded) Catches permanent errors without being user-configurable in v1
Backoff strategy Exponential (100ms, 2x, 30s cap) Not user-configurable; suitable for most transient failures
Status update retry Fixed 1s × 3 attempts Best-effort; uses fire-and-forget semantics

Testing

All tests pass: ./scripts/test-all-local.sh

@pinodeca
Pino de Candia (pinodeca) force-pushed the pinodeca/retries branch 2 times, most recently from 33c1c73 to 779493c Compare March 9, 2026 14:47
- Add pg_durable.max_retries GUC (default 3, reloadable via SIGHUP)
- Use duroxide schedule_activity_with_retry for SQL and HTTP nodes
- Add retry with fixed backoff for status update activities
- Capture max_retries at df.start() time for deterministic replay
- Implement loop resilience: failed iterations are absorbed, loop continues
- Add LoopMetrics tracking (successes, failures, consecutive failures)
- Terminate loop after 10 consecutive failures (safety limit)
- Add E2E tests for node retry and loop resilience
- Update USER_GUIDE.md and api-reference.md with retry documentation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant