[FEAT] Improve Python SDK reliability, retries, and failure diagnostics

[TheEleventhAvatar]

The Python SDK can currently surface generic or low-context errors when encountering
common failure scenarios such as server unavailability, timeouts, or partial workflow
registration failures. In both local development and production environments, this
can make it difficult for users to quickly determine whether an issue is caused by
connectivity, configuration problems, or transient infrastructure issues.

I’d like to improve the reliability and debuggability of the Python SDK by making
these failure modes more explicit and easier to reason about. This would include
centralizing request error handling, introducing clearer and more actionable
exceptions for common scenarios, adding configurable timeout and retry behavior
where appropriate, and improving healthcheck robustness while preserving backward
compatibility with existing SDK usage.

While documenting common failure cases is an alternative, improving behavior directly
in the SDK provides better defaults and faster feedback for users running Hatchet
workflows in production. I’m happy to approach this incrementally and align on scope
before implementation, and wanted to confirm that this is an area where contributions
and potential longer-term ownership would be welcome.

[mrkaye97]

hey @TheEleventhAvatar, would love to see a detailed sketch of your vision here before implementation to get a sense for what you’re thinking and make sure we’re aligned, but definitely agree with the broader goal!

[TheEleventhAvatar]

That sounds good, happy to sketch it out.

At a high level, my thinking is to keep this incremental and non-invasive, focusing
first on making existing failure modes easier to understand rather than changing
behavior.

The initial scope I have in mind would be:

• Introduce a small, explicit exception hierarchy for the Python SDK (e.g. connection
  errors, timeouts, unexpected server responses) to replace generic exceptions while
  preserving existing control flow.
• Centralize request execution and error translation in a single place, so retries,
  timeouts, and diagnostics aren’t scattered across the SDK.
• Add conservative, opt-in timeout and retry defaults that can be configured, but do
  not change behavior for existing users unless explicitly enabled.
• Improve healthcheck failure reporting to clearly distinguish between connectivity
  issues and server-side readiness.

I’d plan to tackle this in small PRs (starting with exception normalization and
centralized request handling), and avoid any breaking changes. If helpful, I can write
up a more concrete proposal or pseudo-code for the first step before starting.

Let me know if this direction aligns with how you’re thinking about it, or if there’s
a specific area you’d prefer to prioritize first.

[mrkaye97]

Sure, great! We have lots of improvements to our documentation that we need, in particular. And I'm also in favor of improving error messages to be more helpful. Happy to take a look if you give it a shot!

Whoops, replied this in the wrong issue.

For this one, let me think about this for a bit and report back!

[mrkaye97]

Okay, so I do think this is a good idea - would be happy to review PRs here. Some of these (e.g. the second) might prove to be more challenging than others, but would definitely like to see improvements on all of these fronts!

[BloggerBust]

I have opened a small, scoped PR that improves Python SDK transport error diagnostics without changing retry behavior.

PR: #2942

This addresses the diagnostics side of the issue; happy to follow up on retries separately if helpful.

[BloggerBust]

I’ve split work on this issue into a few small, incremental PRs:

• PR feat(sdk): improve python REST client transport error diagnostics #2942: improved Python REST client transport error diagnostics
• PR test(rest): add typed REST transport exceptions + diagnostics tests #2968: typed REST transport exceptions + improved diagnostics (no retry behavior changes)
• PR test(sdk): cover python REST and gRPC retry predicate behavior #2971: tests covering the current Tenacity retry predicate for REST + gRPC (documents existing behavior only)

The tests show the current policy:

• REST: retries ServiceException (5xx) and NotFoundException (404)
• gRPC: retries most errors except a small set of clearly permanent status codes
• REST transport errors (timeouts / connection / TLS / protocol) are not retried

Before proposing any behavior changes, such as whether or not 404s should be retried, whether transport errors should be retried, or how to handle 429, could you share preferred retry policy direction for the SDK? Especially for mutating operations where retries may duplicate side effects?

Happy to follow up with a small PR that aligns with whatever policy you prefer.

[mrkaye97]

Thanks for the follow up! I think the current behavior is mostly correct with a couple caveats:

1. We probably should be retrying timeouts, connection issues, etc.
2. We ideally would have retry behavior be more configurable. I'm not too sure what this looks like yet (there's a bit of a hack to allow retries to be shut off via the client config, but that's it), but it'd be better to allow the client to have more granular control, if possible.

For mutating operations, we probably should not retry in most cases, although I'm wary of breaking changes here - there's more work to do on idempotency on the backend to make this safer.

Let me know if you have more questions here! Happy to discuss, lots to consider

[BloggerBust]

Thank you @mrkaye97!

I can follow up with a small, conservative PR that retries REST transport errors without changing semantics for mutating operations.

Proposed scope for a first step:

• Retry RestTransportError, but only for clearly idempotent requests, such as GET.
• Leave current behavior unchanged for mutating ops for now.
• Add focused tests that assert GET is retried while mutating verbs such as POST are not.

Before I begin, a couple quick contract questions to make sure this lines up with your expectations:

1. Are there any non GET endpoints you would want treated as retryable, such as DELETE which is typically idepotent by nature, or should we strictly limit the initial change to GET until backend idempotency guarantees are stronger?

2. On configurability: since we already have TenacityConfig and a minimal retry hook, would you prefer:

   • keeping this PR non-configurable with retries having fixed behavior, and layering configurability later, or
   • starting to extend existing config in a small way as part of this change?