Our current error handling should be improved to handle failures more gracefully. We currently retry failed attempts endlessly and we also endlessly retry Gitlab operations (update comment, etc...) . Adding some backoff with a max attempts would be a good start. We could also augment our Queue messages to maintain a count of attempts to prevent endless retries.
For NATS we could use some combo of these to manage message delivery:
https://pkg.go.dev/github.com/nats-io/nats.go#MaxDeliver
https://pkg.go.dev/github.com/nats-io/nats.go#Msg.Term
https://pkg.go.dev/github.com/nats-io/nats.go#Msg.Metadata
@sl1pm4t pointed to this https://choria.io/blog/post/2020/04/03/nats_patterns_9/ under Redelivery as a way of handling these types of messages.
Our current error handling should be improved to handle failures more gracefully. We currently retry failed attempts endlessly and we also endlessly retry Gitlab operations (update comment, etc...) . Adding some backoff with a max attempts would be a good start. We could also augment our Queue messages to maintain a count of attempts to prevent endless retries.
For NATS we could use some combo of these to manage message delivery:
https://pkg.go.dev/github.com/nats-io/nats.go#MaxDeliver
https://pkg.go.dev/github.com/nats-io/nats.go#Msg.Term
https://pkg.go.dev/github.com/nats-io/nats.go#Msg.Metadata
@sl1pm4t pointed to this https://choria.io/blog/post/2020/04/03/nats_patterns_9/ under Redelivery as a way of handling these types of messages.