Skip to content

Commit 66c794a

Browse files
NiveditJainNiveditJain
committed
Add retry policy documentation and integrate into graph configuration
- Introduced a new documentation file for the Retry Policy feature, detailing its configuration and usage within Exosphere. - Updated the `create-graph.md` file to include a section on retry policies, explaining their structure and providing examples. - Modified `mkdocs.yml` to include the new Retry Policy documentation in the navigation, enhancing accessibility for users.
1 parent f6a3bb2 commit 66c794a

3 files changed

Lines changed: 302 additions & 1 deletion

File tree

‎docs/docs/exosphere/create-graph.md‎

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,13 @@ One can define a graph on Exosphere through a simple json config, which specifie
5151
},
5252
"next_nodes": []
5353
}
54-
]
54+
],
55+
"retry_policy": {
56+
"max_retries": 3,
57+
"strategy": "EXPONENTIAL",
58+
"backoff_factor": 2000,
59+
"exponent": 2
60+
}
5561
}
5662
```
5763

@@ -126,6 +132,23 @@ Use the `${{ ... }}` syntax to map outputs from previous nodes:
126132
- **`${{ node_identifier.outputs.field_name }}`**: Maps output from a specific node
127133
- **`initial`**: Static value provided when the graph is triggered
128134
- **Direct values**: String values. In v1, numbers/booleans must be string-encoded (e.g., "42", "true").
135+
136+
### Retry Policy
137+
138+
Graphs can include a retry policy to handle transient failures automatically. The retry policy is configured at the graph level and applies to all nodes within the graph.
139+
140+
```json
141+
{
142+
"retry_policy": {
143+
"max_retries": 3,
144+
"strategy": "EXPONENTIAL",
145+
"backoff_factor": 2000,
146+
"exponent": 2
147+
}
148+
}
149+
```
150+
151+
For detailed information about retry policies, including all available strategies and configuration options, see the [Retry Policy](retry-policy.md) documentation.
129152
## Creating Graph Templates
130153

131154
The recommended way to create graph templates is using the Exosphere Python SDK, which provides a clean interface to the State Manager API.

‎docs/docs/exosphere/retry-policy.md‎

Lines changed: 276 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,276 @@
1+
# Retry Policy
2+
3+
!!! beta "Beta Feature"
4+
Retry Policy is currently available in beta. The API and functionality may change in future releases.
5+
6+
The Retry Policy feature in Exosphere provides sophisticated retry mechanisms for handling transient failures in your workflow nodes. When a node execution fails, the retry policy automatically determines when and how to retry the execution based on configurable strategies.
7+
8+
## Overview
9+
10+
Retry policies are configured at the graph level and apply to all nodes within that graph. When a node fails with an error, the state manager automatically creates a retry state with a calculated delay before the next execution attempt.
11+
12+
## Configuration
13+
14+
Retry policies are defined in your graph template configuration:
15+
16+
```json
17+
{
18+
"secrets": {
19+
"api_key": "your-api-key"
20+
},
21+
"nodes": [
22+
{
23+
"node_name": "MyNode",
24+
"namespace": "MyProject",
25+
"identifier": "my_node",
26+
"inputs": {
27+
"data": "initial"
28+
},
29+
"next_nodes": []
30+
}
31+
],
32+
"retry_policy": {
33+
"max_retries": 3,
34+
"strategy": "EXPONENTIAL",
35+
"backoff_factor": 2000,
36+
"exponent": 2
37+
}
38+
}
39+
```
40+
41+
## Parameters
42+
43+
### max_retries
44+
- **Type**: `int`
45+
- **Default**: `3`
46+
- **Description**: The maximum number of retry attempts before giving up
47+
- **Constraints**: Must be >= 0
48+
49+
### strategy
50+
- **Type**: `string`
51+
- **Default**: `"EXPONENTIAL"`
52+
- **Description**: The retry strategy to use for calculating delays
53+
- **Options**: See [Retry Strategies](#retry-strategies) below
54+
55+
### backoff_factor
56+
- **Type**: `int`
57+
- **Default**: `2000` (2 seconds)
58+
- **Description**: The base delay factor in milliseconds
59+
- **Constraints**: Must be > 0
60+
61+
### exponent
62+
- **Type**: `int`
63+
- **Default**: `2`
64+
- **Description**: The exponent used for exponential strategies
65+
- **Constraints**: Must be > 0
66+
67+
## Retry Strategies
68+
69+
Exosphere supports three main categories of retry strategies, each with jitter variants to prevent thundering herd problems.
70+
71+
### Exponential Strategies
72+
73+
Exponential strategies increase the delay exponentially with each retry attempt.
74+
75+
#### EXPONENTIAL
76+
Standard exponential backoff without jitter.
77+
78+
**Formula**: `backoff_factor * (exponent ^ retry_count)`
79+
80+
**Example**:
81+
- Retry 1: 2000ms (2 seconds)
82+
- Retry 2: 4000ms (4 seconds)
83+
- Retry 3: 8000ms (8 seconds)
84+
85+
#### EXPONENTIAL_FULL_JITTER
86+
Exponential backoff with full jitter (random delay between 0 and calculated delay).
87+
88+
**Formula**: `random(0, backoff_factor * (exponent ^ retry_count))`
89+
90+
**Example**:
91+
- Retry 1: 0-2000ms (random)
92+
- Retry 2: 0-4000ms (random)
93+
- Retry 3: 0-8000ms (random)
94+
95+
#### EXPONENTIAL_EQUAL_JITTER
96+
Exponential backoff with equal jitter (random delay around half the calculated delay).
97+
98+
**Formula**: `(backoff_factor * (exponent ^ retry_count)) / 2 + random(0, (backoff_factor * (exponent ^ retry_count)) / 2)`
99+
100+
**Example**:
101+
- Retry 1: 1000-2000ms (random)
102+
- Retry 2: 2000-4000ms (random)
103+
- Retry 3: 4000-8000ms (random)
104+
105+
### Linear Strategies
106+
107+
Linear strategies increase the delay linearly with each retry attempt.
108+
109+
#### LINEAR
110+
Standard linear backoff without jitter.
111+
112+
**Formula**: `backoff_factor * retry_count`
113+
114+
**Example**:
115+
- Retry 1: 2000ms (2 seconds)
116+
- Retry 2: 4000ms (4 seconds)
117+
- Retry 3: 6000ms (6 seconds)
118+
119+
#### LINEAR_FULL_JITTER
120+
Linear backoff with full jitter.
121+
122+
**Formula**: `random(0, backoff_factor * retry_count)`
123+
124+
**Example**:
125+
- Retry 1: 0-2000ms (random)
126+
- Retry 2: 0-4000ms (random)
127+
- Retry 3: 0-6000ms (random)
128+
129+
#### LINEAR_EQUAL_JITTER
130+
Linear backoff with equal jitter.
131+
132+
**Formula**: `(backoff_factor * retry_count) / 2 + random(0, (backoff_factor * retry_count) / 2)`
133+
134+
**Example**:
135+
- Retry 1: 1000-2000ms (random)
136+
- Retry 2: 2000-4000ms (random)
137+
- Retry 3: 3000-6000ms (random)
138+
139+
### Fixed Strategies
140+
141+
Fixed strategies use a constant delay for all retry attempts.
142+
143+
#### FIXED
144+
Standard fixed delay without jitter.
145+
146+
**Formula**: `backoff_factor`
147+
148+
**Example**:
149+
- Retry 1: 2000ms (2 seconds)
150+
- Retry 2: 2000ms (2 seconds)
151+
- Retry 3: 2000ms (2 seconds)
152+
153+
#### FIXED_FULL_JITTER
154+
Fixed delay with full jitter.
155+
156+
**Formula**: `random(0, backoff_factor)`
157+
158+
**Example**:
159+
- Retry 1: 0-2000ms (random)
160+
- Retry 2: 0-2000ms (random)
161+
- Retry 3: 0-2000ms (random)
162+
163+
#### FIXED_EQUAL_JITTER
164+
Fixed delay with equal jitter.
165+
166+
**Formula**: `backoff_factor / 2 + random(0, backoff_factor / 2)`
167+
168+
**Example**:
169+
- Retry 1: 1000-2000ms (random)
170+
- Retry 2: 1000-2000ms (random)
171+
- Retry 3: 1000-2000ms (random)
172+
173+
## Usage Examples
174+
175+
### Basic Exponential Retry
176+
```json
177+
{
178+
"retry_policy": {
179+
"max_retries": 3,
180+
"strategy": "EXPONENTIAL",
181+
"backoff_factor": 1000,
182+
"exponent": 2
183+
}
184+
}
185+
```
186+
187+
### Aggressive Retry with Jitter
188+
```json
189+
{
190+
"retry_policy": {
191+
"max_retries": 5,
192+
"strategy": "EXPONENTIAL_FULL_JITTER",
193+
"backoff_factor": 500,
194+
"exponent": 3
195+
}
196+
}
197+
```
198+
199+
### Conservative Linear Retry
200+
```json
201+
{
202+
"retry_policy": {
203+
"max_retries": 2,
204+
"strategy": "LINEAR",
205+
"backoff_factor": 5000
206+
}
207+
}
208+
```
209+
210+
### Fixed Retry for Rate Limiting
211+
```json
212+
{
213+
"retry_policy": {
214+
"max_retries": 10,
215+
"strategy": "FIXED_EQUAL_JITTER",
216+
"backoff_factor": 1000
217+
}
218+
}
219+
```
220+
221+
## When Retries Are Triggered
222+
223+
Retries are automatically triggered when:
224+
225+
1. A node execution fails with an error
226+
2. The current retry count is less than `max_retries`
227+
3. The state status is `QUEUED` or `EXECUTED`
228+
229+
The retry mechanism:
230+
- Creates a new state with `retry_count` incremented by 1
231+
- Sets `enqueue_after` to the current time plus the calculated delay
232+
- Sets the original state status to `ERRORED` with the error message
233+
234+
## Best Practices
235+
236+
### Choose the Right Strategy
237+
- **EXPONENTIAL**: Best for most transient failures (network issues, temporary service unavailability)
238+
- **LINEAR**: Good for predictable, consistent delays
239+
- **FIXED**: Useful for rate limiting scenarios
240+
241+
### Use Jitter for High Concurrency
242+
- **FULL_JITTER**: Best for high concurrency to prevent thundering herd
243+
- **EQUAL_JITTER**: Good balance between predictability and randomization
244+
- **No Jitter**: Use only when you need deterministic behavior
245+
246+
### Set Appropriate Limits
247+
- **max_retries**: Consider the nature of your failures and downstream dependencies
248+
- **backoff_factor**: Balance between responsiveness and resource usage
249+
- **exponent**: Higher values create more aggressive backoff
250+
251+
### Monitor Retry Patterns
252+
- Track retry counts in your monitoring system
253+
- Set up alerts for graphs with high retry rates
254+
- Analyze retry patterns to identify systemic issues
255+
256+
## Limitations
257+
258+
- Retry policies apply to all nodes in a graph uniformly
259+
- Individual node-level retry policies are not supported
260+
- Retry delays are calculated in milliseconds
261+
- Maximum delay is not capped (consider using reasonable `backoff_factor` and `exponent` values)
262+
263+
## Error Handling
264+
265+
If a retry policy configuration is invalid:
266+
- The graph template validation will fail
267+
- An error will be returned during graph creation
268+
- The graph will not be saved until the configuration is corrected
269+
270+
## Integration with Signals
271+
272+
Retry policies work alongside Exosphere's signal system:
273+
274+
- Nodes can still raise `PruneSignal` to stop retries immediately
275+
- Nodes can raise `ReQueueAfterSignal` to re-queue after sometime, this will not mark nodes as failure.
276+
- The retry count is preserved when using signals

‎docs/mkdocs.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,7 @@ plugins:
101101
- exosphere/register-node.md
102102
- exosphere/create-runtime.md
103103
- exosphere/create-graph.md
104+
- exosphere/retry-policy.md
104105
- exosphere/trigger-graph.md
105106
- exosphere/dashboard.md
106107
- exosphere/signals.md
@@ -129,6 +130,7 @@ nav:
129130
- Register Node: exosphere/register-node.md
130131
- Create Runtime: exosphere/create-runtime.md
131132
- Create Graph: exosphere/create-graph.md
133+
- Retry Policy: exosphere/retry-policy.md
132134
- Trigger Graph: exosphere/trigger-graph.md
133135
- Dashboard: exosphere/dashboard.md
134136
- Signals: exosphere/signals.md

0 commit comments

Comments
 (0)