[V2] Backgroud goroutine to persist action state

[machichima]

We just just call run service directly to update the action state to DB when we get update in k8s/client.go.

It would be more robust to add a background goroutine to query the run service for persisting the update, which we can handle retry there

Originally posted by @machichima in #6902 (comment)

[runllm]

My understanding is that Flyte currently updates the action state in the database by calling the run service directly when an update is received in k8s/client.go. There is a suggestion to improve robustness by introducing a background goroutine that would periodically query the run service and persist the update, allowing for retries in case of transient failures. This approach is discussed but not yet implemented, as noted in the referenced GitHub discussion. The idea is to decouple immediate DB writes from event handling, improving reliability in the face of errors or network issues.

It seems likely that this enhancement would involve adding a retry mechanism, similar to existing retry patterns in Flyte's async components, to ensure eventual consistency of action state persistence. However, the exact implementation details or code for such a background goroutine are not present in the provided data sources, only the discussion and intent are mentioned. Would you like more detail on how retries are generally handled in Flyte or how background processing is structured?

Sources:

• Implement state service on top of etcd. #6902 (comment)

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!

[machichima]

Fixing in:

• [V2] Migrate action service #6949
• [V2] Add Internal run service #6956
• [V2][Feat] Add events service #6970