fix(producer): treat kafka_storage_error as retriable

[whatyouhide]

Hey all! Andrea from the Elixir core team here.

We just started using brod (under broadway_kafka) at Knock, for communicating with WarpStream.

This PR adds ?kafka_storage_error (KAFKA_STORAGE_ERROR, error code 56) to brod_producer:is_retriable/1, so a produce response carrying it nacks the buffer and schedules a retry (same path as request_timed_out) instead of exiting the producer with {not_retriable, _}.

The idea is that the Kafka protocol classifies error 56 as Retriable=True ("Disk error when trying to access log file on the disk"; see the protocol error table). For example, the Java client automatically retries it on Produce. brod currently treats it as fatal.

The failure mode is worse than a plain error: exit({not_retriable, _}) kills the producer and loses its whole internal buffer, so a transient broker-side storage blip becomes message loss for any caller that doesn't wrap brod in its own retry layer.

We noticed this because we use WarpStream (Kafka-compatible, S3-backed), which uses KAFKA_STORAGE_ERROR as its generic "back off and retry" code. Their docs explicitly call it "a retriable error code" and several changelog entries pick error 56 precisely because compliant clients treat it as transient.

I'm opening this because in production we observed a single WarpStream-side blip take down a bunch of brod producers (across 12 pods within 15ms 😮), all with {not_retriable, {produce_response_error, Topic, Partition, -1, kafka_storage_error}}.

[mikpe]

LGTM

[zmstone]

thank you for the pr.

• tagged 4.5.6
• published to hex.pm