Implement retry policy and enhance errored state handling (#326)

* Implement retry policy and enhance errored state handling

- Added a retry policy model to manage state retries with configurable methods (fixed, linear, exponential).
- Updated the errored state function to create a retry state if the maximum retries have not been reached, improving error recovery.
- Enhanced the ErroredResponseModel to include a flag indicating whether a retry state was created.
- Modified the GraphTemplate and State models to incorporate retry policy attributes, ensuring better state management.
- Improved validation and error handling in the upsert_graph_template function to accommodate the new retry policy structure.

* Refactor retry policy and errored state handling

- Introduced a new RetryStrategy enum with additional strategies for retrying operations, enhancing flexibility in retry mechanisms.
- Updated the RetryPolicyModel to include a compute_delay method for calculating delays based on the selected strategy.
- Refactored the errored_state function to utilize the new retry policy structure, improving error handling and state management.
- Removed the previous _calculate_enqueue_after function, streamlining the code and enhancing clarity.

* Add retry policy documentation and integrate into graph configuration

- Introduced a new documentation file for the Retry Policy feature, detailing its configuration and usage within Exosphere.
- Updated the `create-graph.md` file to include a section on retry policies, explaining their structure and providing examples.
- Modified `mkdocs.yml` to include the new Retry Policy documentation in the navigation, enhancing accessibility for users.

* Enhance retry policy error handling and validation

- Added validation to ensure the retry count is greater than 0 in the compute_delay method of RetryPolicyModel, raising a ValueError for invalid inputs.
- Updated the compute_delay method to correctly calculate delays based on the retry count, adjusting the exponentiation logic.
- Refined error handling in the GraphTemplate model by replacing ValueError with HTTPException for better integration with FastAPI, ensuring a 404 response when a graph template is not found.

* Update retry policy documentation and examples

- Clarified the backoff_factor parameter in the retry policy documentation to specify its unit as milliseconds.
- Added retry policy examples in the create-graph.md file to demonstrate its usage in graph template creation and updates.
- Ensured consistency in the retry policy structure across documentation, enhancing user understanding and implementation.

* Enhance retry policy implementation and documentation

- Added the `max_delay` parameter to the retry policy model, allowing users to cap the maximum delay for retry attempts.
- Updated the documentation to include detailed explanations of the `max_delay` parameter and its usage in retry strategies.
- Improved error handling in the `errored_state` function to log errors when fetching graph templates and raise appropriate HTTP exceptions.
- Refactored the `GraphTemplate` model to raise a ValueError instead of an HTTPException when a graph template is not found, enhancing error handling consistency.
- Updated the `compute_delay` method in the `RetryPolicyModel` to apply the new delay capping logic across all retry strategies.

* Enhance errored state handling with retry state management

- Added error handling for duplicate retry states in the `errored_state` function, logging when a retry state already exists.
- Introduced a new `fanout_id` field in the `State` model to support unique identification of retry states.
- Updated the database index to enforce uniqueness on the combination of relevant state fields, improving data integrity and query performance.

* Update state-manager/app/models/db/state.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update docs/docs/exosphere/retry-policy.md

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update docs/docs/exosphere/retry-policy.md

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update state-manager/app/controller/errored_state.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update state-manager/app/controller/errored_state.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update state-manager/app/models/retry_policy_model.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update max_delay description in RetryPolicyModel to clarify behavior when set to None

* Refine documentation for retry policy and errored state handling

- Added missing newlines in the retry policy documentation for better readability.
- Removed redundant assignment in the errored state handling to streamline the code logic.

* Enhance tests for errored state and upsert graph template

- Added retry policy integration in the test cases for errored state handling and upsert graph template.
- Updated test fixtures to include additional state attributes for better simulation of state behavior.
- Improved error handling assertions in the errored state tests to ensure proper HTTP exceptions are raised.
- Mocked retry policy in graph template tests to validate retry behavior during upsert operations.

* Refactor test for RetryPolicyModel by removing unnecessary import

- Removed the unused 'random' import from the test file for cleaner code.
- Updated the instantiation of RetryPolicyModel to improve readability.

* Add comprehensive tests for errored state handling in graph templates

- Introduced multiple test cases to cover scenarios where graph templates are not found, encounter other errors, or trigger DuplicateKeyError during state creation.
- Enhanced error handling assertions to ensure proper HTTP exceptions are raised for missing graph templates and other exceptions.
- Validated behavior when maximum retries are reached, ensuring no new state is created in such cases.
- Improved overall test coverage for errored state functionality.

* Refactor assertions in errored state tests for clarity

- Updated assertions in the TestErroredState class to use 'not' instead of '== False' for improved readability.
- Ensured consistency in the test code style across multiple test cases.

* Remove Kubernetes deployment steps from the publish workflow

- Eliminated the deploy-to-k8s job from the publish-state-manager workflow to streamline the CI/CD process.
- This change focuses on publishing the image without the deployment step, simplifying the workflow configuration.

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Author: NiveditJain

.github/workflows/publish-state-mangaer.yml

@@ -111,30 +111,3 @@ jobs:
           push: true
           tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}
-
-  deploy-to-k8s:
-    needs: publish-image
-    runs-on: ubuntu-latest
-
-    steps:
-      - name: Deploy to K8s
-        run: |
-          echo "${{ secrets.KUBE_CONFIG }}" | base64 -d > kubeconfig.yaml
-          export KUBECONFIG=$PWD/kubeconfig.yaml
-          kubectl get nodes
-
-          echo "selected image: ${{ fromJson(needs.publish-image.outputs.json).tags[1] }}"
-
-          kubectl set image deployment/exosphere-state-manager exosphere-state-manager=${{fromJson(needs.publish-image.outputs.json).tags[1]}}
-
-          kubectl rollout status deployment/exosphere-state-manager
-
-          status=$(kubectl rollout status deployment/exosphere-state-manager)
-
-          echo "$status"
-
-          if [[ "$status" != *"successfully rolled out"* ]]; then
-            kubectl rollout undo deployment/exosphere-state-manager
-            echo "❌ Deployment failed. Rolled back." >&2
-            exit 1
-          fi

docs/docs/exosphere/create-graph.md

@@ -51,7 +51,13 @@ One can define a graph on Exosphere through a simple json config, which specifie
       },
       "next_nodes": []
     }
-  ]
+  ],
+  "retry_policy": {
+    "max_retries": 3,
+    "strategy": "EXPONENTIAL",
+    "backoff_factor": 2000,
+    "exponent": 2
+  }
 }
 ```
 
@@ -126,6 +132,24 @@ Use the `${{ ... }}` syntax to map outputs from previous nodes:
 - **`${{ node_identifier.outputs.field_name }}`**: Maps output from a specific node
 - **`initial`**: Static value provided when the graph is triggered
 - **Direct values**: String values. In v1, numbers/booleans must be string-encoded (e.g., "42", "true").
+
+### Retry Policy
+
+Graphs can include a retry policy to handle transient failures automatically. The retry policy is configured at the graph level and applies to all nodes within the graph.
+
+```json
+{
+  "retry_policy": {
+    "max_retries": 3,
+    "strategy": "EXPONENTIAL",
+    "backoff_factor": 2000, // milliseconds
+    "exponent": 2
+  }
+}
+```
+
+For detailed information about retry policies, including all available strategies and configuration options, see the [Retry Policy](retry-policy.md) documentation.
+
 ## Creating Graph Templates
 
 The recommended way to create graph templates is using the Exosphere Python SDK, which provides a clean interface to the State Manager API.
@@ -156,7 +180,13 @@ async def create_graph_template():
         result = await state_manager.upsert_graph(
             graph_name="my-workflow",
             graph_nodes=graph_nodes,
-            secrets=secrets
+            secrets=secrets,
+            retry_policy={
+                "max_retries": 3,
+                "strategy": "EXPONENTIAL",
+                "backoff_factor": 2000,
+                "exponent": 2
+            }
         )
         print("Graph template created successfully!")
         print(f"Validation status: {result['validation_status']}")
@@ -268,7 +298,13 @@ The state manager validates your graph template:
             result = await state_manager.upsert_graph(
                 graph_name="my-workflow",
                 graph_nodes=updated_nodes,
-                secrets=updated_secrets
+                secrets=updated_secrets,
+                retry_policy={
+                    "max_retries": 3,
+                    "strategy": "EXPONENTIAL",
+                    "backoff_factor": 2000,
+                    "exponent": 2
+                }
             )
             print("Graph template updated successfully!")
             print(f"Validation status: {result['validation_status']}")